When are these two subgroups equal?

Question

Suppose $H$ is a subgroup of $G$ and let $X$ be the set of left cosets $xH$ of $H$ in $G$.

I have shown that the kernel of the action $g\cdot xH=gxH, $ $K$ is a normal subgroup in $G$ that is contained in $H$. My question is:

Under what conditions do we have $K=H$?

I have found that $\displaystyle K=\bigcap_{xH\in X} xHx^{-1}$ so need to determine when $\displaystyle H=\bigcap_{xH\in X} xHx^{-1}$

Answer 1

This happens if and only if $H$ is normal. Since $K$ is normal if $K = H$ then clearly $H$ is normal. Conversely, assume $H$ is normal. Then for any $h \in H$ we have $hxH = x(x^{-1}hx)H = xH$, where the last equality is because $H$ is normal so $x^{-1}hx \in H$. This proves that $H \subseteq K$. You've already shown $K \subseteq H$ so $K = H$.

Answer 2

Theorem. The kernel $K$ of the action is the core of $H$ in $G$, that is, the largest normal subgroup of $G$ that is contained in $H$.

Proof. Note that there is such as a thing as "the largest normal subgroup of $G$ that is contained in $H$": for if $\{N_i\}_{i\in I}$ is any family of normal subgroups contained in $H$, then $N=\langle N_i\mid i\in I\rangle$ is a subgroup of $G$ contained in $H$, and is normal: for given any $n\in\cup_{i\in I}N_i$, and $g\in G$, then $gng^{-1}\in\cup_{i\in I}N_i$. Since $\cup_{i\in I}N_i$ generates $N$, this proves that $N$ is normal. Thus, taking the subgroup generated by all normal subgroups contained in $H$ yields the largest normal subgroup of $G$ contained in $H$.

So let $N$ be the core of $H$ in $G$. Trivially, since $K$ is a normal subgroup of $G$ contained in $H$, we have $K\subseteq N$.

Conversely, let $n\in N$. To show that $n$ is in the kernel of the action, let $xH$ be a left coset. Then $n(xH) = nxH = x(x^{-1}nx)H = xH$ (since $x^{-1}nx\in N\subseteq H$). Thus, $n$ fixes all cosets. Hence, $N\subseteq K$, proving equality. $\Box$

The above gives the "bottoms up" construction of the core; the intersection definition you give yields the "top down" construction of the core. (See this post for a general discussion of bottoms-up vs. top-down constructions).

In particular, when is "the largest normal subgroup of $G$ contained in $H$" equal to $H$? Precisely when $H$ is itself normal, as noted by Jim.