Let $so(3)$ be the Lie Algebra of $SO(3)$ and $R\in SO(3); \Omega_1,\Omega_2 \in so(3)$ and $\Omega_n = \frac{d}{dt}R_n(t)$ at the point $t=0$. So $\Omega_n$ is the tangent vector of the curve $R_n(t)$ at $t=0$.
To show that the Lie bracket $[\Omega_1,\Omega_2]$ is an element of $so(3)$ my professor wrote:
$$ \tag{1}[\Omega_1,\Omega_2] = \left.\frac{d}{dt}R_1(t)\Omega_2 R_1(t)^{-1}\right|_{t=0} $$
I am trying to understand that, but I cannot see why this shows that the commutator is an element of $so(3)$.
I understand that for this tangential vector to be in $so(3)$, the product $R_1(t)\Omega_2 R_1(t)^{-1}$ has to be a parametrized curve in $SO(3)$.
But I do not see why it is a curve in $SO(3)$.
We want to show that $$ \tag{1}[\Omega_1,\Omega_2] = \left.\frac{d}{dt}R_1(t)\Omega_2 R_1(t)^{-1}\right|_{t=0} $$
We can start doing this by computing: \begin{eqnarray*} \left.\frac{d}{dt}\left(R_1(t)\Omega_2 R_1(t)^{-1}\right)\right|_{t=0} &=& \left.\left[\frac{d}{dt}R_1(t)(\Omega_2 R_1(t)^{-1})\right]\right|_{t=0} + \left.\left[(R_1(t)\Omega_2) \frac{d}{dt}R_1(t)^{-1}\right]\right|_{t=0} \\ &=& \left[\Omega_1\Omega_2 R_1(0)^{-1}\right] + \left.\left[R_1(t)\Omega_2 \frac{d}{dt}R_1(t)^{-1}\right]\right|_{t=0} \\ &=& \left[\Omega_1\Omega_2 R_1(0)^{-1}\right] + \left.\left[R_1(t)\Omega_2 (-R_1(t)^{-1})\left(\frac{d}{dt}R_1(t)\right)R_1(t)^{-1})\right]\right|_{t=0} \\ &=& \left[\Omega_1\Omega_2 R_1(0)^{-1}\right] - \left.\left[R_1(t)\Omega_2 R_1(t)^{-1}\left(\frac{d}{dt}R_1(t)\right)R_1(t)^{-1})\right]\right|_{t=0} \\ &=& \left[\Omega_1\Omega_2 R_1(0)^{-1}\right] - \left[R_1(0)\Omega_2 R_1(0)^{-1}\left.\left(\frac{d}{dt}R_1(t)\right|_{t=0}\right)R_1(0)^{-1})\right] \\ &=& \left[\Omega_1\Omega_2 R_1(0)^{-1}\right] - \left[R_1(0)\Omega_2 R_1(0)^{-1}\Omega_1R_1(0)^{-1})\right] \\ \end{eqnarray*}
See Derivatives of Inverse Matrix for details on the third line's derivation.
There isn't much further we can go without knowing more details about $R_1$. I actually think it might have been defined as $$R_i(t) = e^{\Omega_it}$$ But it's only necessary to require $R_i(0) = I$. Then the final line above gives $\Omega_1\Omega_2 - \Omega_2\Omega_1$.
Update to include additional requirements:
This is not always true. A Lie group is a subgroup of $GL$ but a Lie algebra always contains the $0$ matrix. It maps to the identity of the Lie group. So if $\Omega_2 = 0$ in the above, then the curve $C(t) = 0$ does not lie in $SO(3)$.
This is true independently of the above considerations. $\Omega_1, \Omega_2$ are elements of $\mathfrak{so}(3)$ which is closed under addition and multiplication (matrices of the form $X^T = -X$).