Let $H$ be a subgroup of $G$, and suppose that $G$ acts by multiplication over the set $X:=G/H$ of the left-hand side classes of $H$ over $G$.

Question

This action is transitive. Now let be $yH \in X$. What is the kernel of this action?

I'm afraid that my answer isn't right. Could you check if the KERNEL is exactly this?

MY ANSWER:

Let $A:G \times X \rightarrow [G:H]=X$ be the action mentioned and $\lambda:G\rightarrow \operatorname{Sym}(G)$ (this homomorphism exists because of the permutational representation). So, we have

$ \begin{align*} \operatorname{ker}(A)&=\{(g,yH) \in A: A(g, yH)=gyH=yH, \forall (g, yH)\}\\ &=\{g \in G: gyH=yH, \forall g \in G\}\\ &=yHy^{-1}, \forall g \in G\; \text{the reason of this step is going to be explained below**}\\ &=\displaystyle\bigcap_{g \in G} yHy^{-1} \end{align*} $

Explanation about **:

$ \begin{align*} \operatorname{Stab}(yH)&=\{g \in G: gyH=yH\}\\ &=\{g \in G: y^{-1} gyH=H\}\\ &=\{g \in G: y^{-1}gy \in H\}\\ &=\{g \in G: g \in yHy^{-1}\}\\ &=yHy^{-1} \end{align*} $

Answer 1

In general, if we are given an action of a group $G$ on a set $X$, $G\times X\to X$, we can establish a homomorphism $\lambda\colon G \to \operatorname{Sym}(X)$ by defining $\lambda_g(x):=g\cdot x$. In fact, $\forall g,h\in G,\forall x\in X$:

$$\lambda_{gh}(x)=(gh)\cdot x=g\cdot(h\cdot x)=\lambda_g(\lambda_h(x))=(\lambda_g\lambda_h)(x)$$

whence:

$$\lambda_{gh}=\lambda_g\lambda_h, \space\forall g,h\in G \tag 1$$

(That indeed $\lambda_g \in \operatorname{Sym}(X), \forall g \in G$, also follows from action's axioms.)

The kernel of this homomorphism (the so-called "kernel of the action") is given by:

\begin{alignat}{1} \operatorname{ker}\lambda &= \{g\in G\mid\lambda_g=\iota_X\} \\ &= \{g\in G\mid\lambda_g(x)=\iota_X(x), \forall x \in X\} \\ &= \{g\in G\mid g\cdot x=x, \forall x \in X\} \\ &= \{g\in G\mid g\in \operatorname{Stab}(x), \forall x \in X\} \\ &= \{g\in \operatorname{Stab}(x), \forall x \in X\} \\ &= \bigcap_{x\in X}\operatorname{Stab}(x) \\ \tag 2 \end{alignat}

In your case, $X=\{gH, g\in G\}$ and $\operatorname{Stab}(gH)=\{g'\in G\mid g'gH=gH\}$.

Lemma. $\operatorname{Stab}(gH)=gHg^{-1}$.

Proof.

\begin{alignat}{1} g'\in \operatorname{Stab}(gH) &\Rightarrow \exists h,h'\in H\mid g'gh=gh' \\ &\Rightarrow \exists h,h'\in H\mid g'=gh'(gh)^{-1}=gh'h^{-1}g^{-1} \\ &\Rightarrow \exists h''\in H\mid g'=gh''g^{-1} \\ &\Rightarrow g'\in gHg^{-1} \\ \end{alignat}

and thence $\operatorname{Stab}(gH)\subseteq gHg^{-1}$. Viceversa:

\begin{alignat}{1} g'\in gHg^{-1} &\Rightarrow g'g \in gH \\ &\Rightarrow g'gH\subseteq gH \\ \tag 3 \end{alignat}

Now:

\begin{alignat}{1} gH\subseteq g'gH &\iff \forall h \in H, \exists h'\in H\mid gh=g'gh' \\ &\iff \forall h \in H, \exists h'\in H\mid gh=(gh''g^{-1})gh'=gh''h' \\ &\iff h'=(gh'')^{-1}gh=h''^{-1}g^{-1}gh=h''^{-1}h \\ \end{alignat}

Since $h''$ exists by hypothesis ("Viceversa..."), such a $h'$ exists, and then indeed:

$$gH\subseteq g'gH \tag 4$$

Therefore, $g'\in gHg^{-1} \Rightarrow g'gH=gH \Rightarrow g'\in \operatorname{Stab}(gH) \Rightarrow gHg^{-1}\subseteq \operatorname{Stab}(gH)$.

$\Box$

By $(2)$ and the Lemma, we have finally:

$$\operatorname{\ker}\lambda = \bigcap_{gH\in G/H}\operatorname{Stab}(gH)= \bigcap_{g\in G}\operatorname{Stab}(gH)=\bigcap_{g\in G}gHg^{-1} \tag 5$$

Answer 2

The kernel of $\lambda$ should be $\bigcap_{g\in G}gHg^{-1}$. This is called the normal core of $H$.

I think this is what you want. $A$ doesn't have a kernel, as $X$ is only a set. But $\operatorname{Sym}X$ is a group, and $\lambda$ a homomorphism. So we can talk about $\operatorname{ker}\lambda:=\{g\in G:\lambda(g)=e\}$.