I want to prove that if $G$ is a connected Lie group, $H$ is a normal Lie subgroup of $G$, $\mathfrak{g}$ and $\mathfrak{h}$ their respective lie algebras, then $\mathfrak{h}$ is an ideal of $\mathfrak{g}$.
I try to proceed as follows:
Let $X\in\mathfrak{g}$, $Y\in\mathfrak{h}$, we want to compute $[X,Y]$ and show it is in $\mathfrak{h}$. Let $g:(-\varepsilon,\varepsilon)\rightarrow G$ and $h:(-\varepsilon,\varepsilon)\rightarrow H$ with $g(0) = h(0) = e$ be such that $$\left.\frac{d}{dt}\right|_{t=0}g(t) = X$$ and $$\left.\frac{d}{ds}\right|_{s=0}h(s) = Y$$ Then $$[X,Y]=\left.\frac{d}{dt}\right|_{t=0}Ad_{g(t)}Y = \left.\frac{d}{dt}\right|_{t=0}\left.\frac{d}{ds}\right|_{s=0}g(t)h(s)g(t)^{-1}$$ Since $H$ is normal in $G$, we have that $g(t)h(s)g(t)^{-1}=\tilde{h}(s,t)\in H$ for all $t,s$. This means that differentiating $\tilde{h}$ once at the origin gives us an element of $\mathfrak{h}$, but here we have to differentiate it twice!
Am I doing something wrong or there is simply a detail I'm not getting?
Hopefully everything is going well.
If $\mathfrak g$ is the Lie algebra of $G$ and $Ad:G\rightarrow GL(\mathfrak g)$ the adjoint representation then $ad:\mathfrak g\rightarrow End(\mathfrak g)$ is its derivative.
So, $[X,Y]=ad_XY=\underbrace{\left(\left.\dfrac{d}{dt}\right|_{t=0} Ad_{e^{tX}} \right)}_{\in End(\mathfrak h)} \quad \underbrace{Y}_{\in \mathfrak h} =\underbrace{\left(\left.\dfrac{d}{dt}\right|_{t=0} Ad_{e^{tX}} \right)}_{\in End(\mathfrak h)} \ \underbrace{(\left.\dfrac{d}{du}\right|_{u=0}e^{uY})}_{\in \mathfrak h} \in \mathfrak h$
The last equality appears to be the same as $\left.\dfrac{d}{dt} \dfrac{d}{du}\right|_{t=u=0}e^{tX}e^{uY}e^{-tX}$ in the special case of $G=GL_n$. Moreover, you can see that we are still working in $\mathfrak h$ and this is how it works for general Lie groups.
By the way, you are computing a Lie bracket of vector fields, and you know that the Lie bracket of two vector fields acting on functions as a derivation involves second derivatives that can cancel out. It is the same here.
For general Lie groups, we can do as follows:
Let $c_g:G\rightarrow G, x\mapsto gxg^{-1}$ be the conjugation map. Then $c_g(H)\subset H$ since $H$ is normal in $G$ and one can consider the restriction of $c_g$ on $H$ into $H$.
Thus $(d c_g)_1(\mathfrak h)=Ad_g(\mathfrak h)\subset \mathfrak h$ and $\forall X\in \mathfrak g, Y\in \mathfrak h$ : $$\forall t, \qquad Ad_{e^{tX}}(Y)\in \mathfrak h\Longrightarrow [X,Y] = ad_XY = \left. \dfrac{d}{dt}\right|_{t=0}Ad_{e^{tX}}(Y) \in \mathfrak h.$$