Is a conjugation of a Lie subgroup a Lie subgroup?
- Conjugation of a subgroup is a subgroup. Thus, a conjugation of a Lie subgroup is a subgroup.
- Conjugation is a diffeomorphism. Thus, a conjugation of a Lie subgroup is the image of a manifold by a diffeomorphism, and so it is an embedded submanifold.
- Finally I need to prove the smoothness of the group multiplication and inverse operation. But I'm not sure that the restrictions of them on the submanifold are still smooth. How can I prove this?
Consider a Lie subgroup $H$ of the Lie group $G$ and fix $g_0\in G$. As you mention, the conjugation map $\varphi\colon h\mapsto g_0hg_0^{-1}$ is a diffeomorphism of $H$ onto its image $H'$. Your question is then whether \begin{equation} \mu'\colon H'\times H' \to H' \end{equation} and \begin{equation} \iota'\colon H'\to H' \end{equation} are smooth. Notice that since $\varphi$ is a group homomorphism, we may write \begin{equation} \iota' = \varphi\circ\iota\circ\varphi', \end{equation} where $\iota$ is the inverse map on $H$. Since the three maps on the right are smooth, so is $\iota'$. Can you say something similar for $\mu'$?