Proving the trace of a representation is equal to zero

Question

I'm having some trouble in beginner's representation theory and am pretty lost about this problem:

Let ($\rho$, $V$) be a representation of $G$, so $\rho$: $G$ $\to$ $GL(V)$ is a group homomorphism. Let $H$$\subset$$G$ be a normal subgroup of index 2. Suppose $V$ is $G$-irreducible, but not $H$-irreducible. Prove tr($\rho$($g$))=0 for $g$$\notin$$H$.

I thought about starting by saying $G/H$ is isomorphic to $C_2$, and so you could say that some character

$\lambda$= \begin{cases} 1, & \text{if $g \in H$} \\ -1, & \text{if $g \notin H$} \end{cases}

But I don't think this is very helpful, and I feel like this is not the way to go about the proof. Any help would be appreciated!

Answer 1

Let $\chi$ be the character of $\rho$. From orthogonality of characters, $$\sum_{g\in G}|\chi(g)|^2=|G|$$ since $\rho$ is irreducible on $G$, and $$\sum_{g\in H}|\chi(g)|^2=m|H|=\frac m2|G|$$ where $m\ge2$ is an integer, since $\rho$ is reducible on $H$. Then $|G|\ge\frac m2 |G|$, so $m=2$ and we have $$\sum_{g\notin H}|\chi(g)|^2=0$$ etc.

Answer 2

Here is an alternative solution that does not directly use character theory. We are told that $H$ does not act irreducibly on $V$, so let $W$ be a nonzero subspace of $V$ of smallest dimension that is invariant under the action of $H$.

Now choose $g \in G \setminus H$. The normality of $H$ in $G$ implies that $g(W)$ is also invariant under that action of $H$. Also, since $g^2 \in H$, $g^2(W) = W$.

Now $W + g(W)$ is invariant under $G$ and so, since $G$ acts irreducibly on $V$, we have $V = W + g(W)$. Also, by minimality of $W$, we must have $W \cap g(W) = \{0\}$, so $V = W \oplus g(W)$.

Now, all elements of $G \setminus H = gH$ interchange the $H$-invariant subspaces $W$ and $g(W)$ so, by choosing a basis of $V$ that consists of a union of bases of $W$ and of $g(W)$, we see that the matrices of the elements of $G \setminus H$ with respect to this basis have trace $0$.