How to prove this index equality problem with group actions?

Question

Let $H$ be a $p$-subgroup of a finite group $G$. Prove that: $[N_G(H):H] \equiv [G:H] (\text{mod} \; p)$. Hint: Consider the appropriate action of $H$ on the set of all cosets of $H$ in $G$.

I literally have no idea what to do. I suppose the action that is meant is the natural one;

$H \times (G / H )\to G/H$, defined as: $a \cdot b H = ab H$.

I guess we could somehow use the class formula: $|G / H| = |Z.| + \sum_{gH \notin Z}[H:H_{gH}]$.

I have no idea what to do with the normaliser: $N_G(H) = \{ a \in G; aHa^{-1}=H \}$ or how to proceed with the problem.

Answer 1

Consider the action of $H$ on $G/H$ by left-multiplication. We know $G/H$ is a disjoint union of orbits of this action, and all the orbits have size dividing $|H|$ (by the orbit-stabilizer theorem). The only orbits that don't have $p$-power size are the fixed points. Therefore, $[G:H]\equiv|\Omega|$ mod $p$, where $\Omega$ is the set of fixed points.

A coset $gH$ is fixed by the action of $H$ if and only if $g\in N_G(H)$ (exercise), so $\Omega=N_G(H)/H$.

Answer 2

Your guess is correct. Consider the action of $H$ by left multiplication on the left quotient set $X:=\{gH, g\in G\}$, namely $(h,gH)\mapsto hgH$. The orbit equation reads: $$|X|=\left|\bigcap_{h\in H}\operatorname{Fix}(h)\right|+\sum_i\frac{|H|}{\left|\operatorname{Stab}(g_iH)\right|}\tag 1$$ where the orbit of $g_iH$ has size greater than $1$, and hence (as $H$ is a $p$-group) equal to some proper power of $p$. Therefore, $p\mid \sum_i\frac{|H|}{\left|\operatorname{Stab}(g_iH)\right|}$, and hence: $$|X|\equiv \left|\bigcap_{h\in H}\operatorname{Fix}(h)\right|\pmod p\tag 2$$ Now, note that: $$|X|=[G:H]\tag 3$$ and \begin{alignat}{1} \bigcap_{h\in H}\operatorname{Fix}(h) &= \{gH\in X\mid hgH=gH, \forall h\in H\} \\ &= \{gH\in X\mid g^{-1}hgH=H, \forall h\in H\} \\ &= \{gH\in X\mid g^{-1}hg\in H, \forall h\in H\} \\ &= \{gH\in X\mid g^{-1}Hg\subseteq H\} \\ &= \{gH\in X\mid g^{-1}Hg=H\} \\ &= \{gH\in X\mid g\in N_G(H)\} \\ &= N_G(H)/H \\ \tag 4 \end{alignat} By $(3)$ and $(4)$, the equation $(2)$ yields: $$[G:H]\equiv [N_G(H):H]\pmod p$$