Let $G$ be a $p$-group and let $H$ be a proper subgroup of $G$. Show that $H$ is a proper subgroup of $N_G(H)$

Question

Let $G$ be a $p$-group and let $H < G$. Show that $H < N_G(H)$.

If $H \trianglelefteq G$, then it should be clear that $H<N_G(H)$. So we suppose, $H$ isn't normal. So we let $H$ act on the set

$$S = \{gHg^{-1}: g \in G, gHg^{-1} \neq H\}$$

via conjugation. If I can show that $p$ doesn't divide $|S|$, then I can use the fixed point theorem to complete the problem. Any advice on how to show that $p$ doesn't divide $|S|$?

Answer 1

The result is only true if $G$ is finite, which you are assuming but never state explicitly. As Derek Holt points out in comment, that is bad practice: if you are considering only finite groups, you should say so (or, in this site, at least tag it as [finite-groups]).

You state that you only need to show that the cardinality of $S$ is not a multiple of $p$ when $H$ is not a normal subgroup of $G$, so that is what the argument below will show.

Since you talk about the fixed point theorem, presumably you know about group actions. Let $G$ act on its subgroups by conjugation. Let $H$ be a proper subgroup of $G$.

Let $T=\{gHg^{-1}\mid g\in G\}$ be the set of all conjugates of $H$. Your set $S$ is just $T\setminus\{H\}$. The set $T$ is the orbit of $H$ under the action.

By the Orbit-Stabilizer Theorem, the cardinality of $T$ is equal to the index of the stabilizer of $H$ under the action. The stabilizer is $$N_G(H) = \{g\in G\mid gHg^{-1}=H\}.$$ Thus, the cardinality of $T$ is $[G:N_G(H)]$, and the cardinality of $S$ is $|T|-1 = [G:N_G(H)] - 1$.

Because $G$ is a $p$-group, every subgroup has order a power of $p$ and index a power of $p$. So the cardinality of $T$ is a power of $p$. Say $p^i$.

That means that the cardinality of $S$ is one less than a power of $p$. If $i\gt 0$, then $|T|\equiv 0\pmod{p}$, so $|S|\equiv -1\pmod{p}$, Since $-1$ is never a multiple of a prime, then it follows that $|S|$ is not a multiple of $p$ and we are done.

If $i=0$, so $|T|=1$, then that means that $gHg^{-1}=H$ for all $g\in G$, so $H\triangleleft G$. Since we are assuming that $H$ is a proper subgroup of $G$, then this gives $H\subsetneq N_G(H)=G$, and we are done without having to worry about $S$ at all.

Answer 2

Hint

1. Finite $p$- groups are nilpotent.
2. For any proper subgroup $H$ of a nilpotent group $G$, we have $N_G(H)\supsetneq H$.

Both these results are well-known.

The result is false without requiring the $p$- group to be finite, as Tarski monsters show.