Let $H$ be a finite-dimensional Lie group and let $G\subset H$ be a closed Lie subgroup thereof. Let us say that $G$ is a core-free Lie subgroup of $H$ if and only if $G$ contains no normal subgroups of $H$ (except, of course, for the trivial group {e} $\subset G\subset H$). I am interested in how one might find all of the core-free Lie subgroups $G$ of a given Lie group $H$.
For instance:
Example 1) If $H$ is abelian, then all of its subgroups are normal. From this, it follows that the only core-free Lie subgroup of $H$ is the trivial group, $G=$ {e}.
Example 2) If $H=\text{SO}(3)$, then two core-free Lie subgroups are the trivial group, $G=$ {e} and $G=\text{SO}(2)$. But are there any others?
Example 3) Suppose that $H=\text{SE}(2)$ is the special Euclidean group (i.e., the group of translations and rotations in the plane without reflections). Just as in the previous example, two core-free Lie subgroups of $H$ are the trivial group, $G=$ {e} and $G=\text{SO}(2)$. In this case, however, there is at least one more. Namely, $G=(\mathbb{R},+)\rtimes\mathbb{Z}_2$ the group of translations in one direction, together with rotation by $\theta=\pi$. But are there any others?
The reason that I care is that (as is well known) taking the quotient of a Lie group $H$ by one of its closed Lie subgroups $G\subset H$ results in a smooth manifold, $\mathcal{M}=H/G$. The left action of $H$ on itself ends up being naturally represented as a group of diffeomorphisms on $\mathcal{M}$, call them $D(H)\subset\text{Diff}(\mathcal{M})$. It turns out that we have $D(H)\cong H$ if and only if $G$ is a core-free Lie subgroup of $H$.
For instance (revisiting the above examples):
Example 2) If $H=\text{SO}(3)$ and $G=\text{SO}(2)$ then we have $\mathcal{M}\cong S^2$ being the two-sphere with $D(H)\cong \text{SO}(3)$ being rigid rotations of this sphere.
Example 3) If $H=\text{SE}(2)$ and $G=\text{SO}(2)$ then we have $\mathcal{M}\cong \mathbb{R}^2$ being the plane with $D(H)\cong \text{SE}(2)$ being rigid translations and rotations of this plane.
Example 3') If $H=\text{SE}(2)$ and $G=(\mathbb{R},+)\rtimes\mathbb{Z}_2$ then we have that $\mathcal{M}=H/G$ is the Mobius strip with $D(H)\cong \text{SE}(2)$ acting on it as a certain collection of diffeomorphisms. (It's a fun puzzle to visualize these.)
I would appreciate any help in characterizing these core-free subgroups either in the $H=\text{SO}(3)$ and $H=\text{SE}(2)$ cases or more generally.
I have made some substantial progress towards answering this question, especially in the $H=\text{SO}(3)$ and $H=\text{SE}(2)$ cases because of their low dimensionality.
In general, given some closed core-free Lie subgroup $G\subset H$ we have that $\mathcal{M}=H/G$ is a smooth homogeneous manifold with $D(H)\cong H$ acting transitively over it. Indeed, because $D(H)$ are diffeomorphisms on $\mathcal{M}$, their action is also faithful.
As I will now discuss, given any such manifold, $\mathcal{M}_0$, I can find a closed core-free Lie subgroup $G_0\subset H$. Suppose that I am given some smooth homogeneous manifold $\mathcal{M}_0$ with a transitive and faithful $H$-action. I can then compute $G_0\subset H$, the stabilizer subgroup of the $H$-action. It is easy to prove that $G_0$ is a closed core-free Lie subgroup of $H$. Hence, the search for closed core-free Lie subgroup of $G_0\subset H$ is exactly the search for smooth homogeneous manifold $\mathcal{M}_0$ with a transitive faithful $H$-action.
Applying this result in the $H=\text{SE}(2)$ we are looking for smooth homogeneous manifolds of dimension three or less which have a transitive and faithful $\text{SE}(2)$-action. But all two-dimensional surfaces on which a (finite-dimensional) Lie group acts transitively are known (see (Mostow 2005)). They are the sphere, the real projective plane, the torus, the plane, the cylinder, the Moebius band, and the Klein bottle. One can check by hand that of these only the plane and the Moebius band have transitive $\text{SE}(2)$ actions.