Let $G$ be a finite group and $N$ a normal subgroup of $G$. Set $Q:=G/N$, the quotient group. Denote by $\rho_G$ the real regular representation of $G,$ ie $\rho_G = \mathbb{R}[G].$ Now consider $\rho_Q \subseteq \rho_G.$ Then we have $\rho_G= \rho_Q \oplus \rho_Q^\perp.$
What are the subgroups $H$ of $G$, such that the fixed point set $(\rho_Q^\perp)^H\neq 0$?
In particular, for $G= C_{p^n},$ cyclic group of order $p^n,$ and $N=C_{p^k}$, (k < n) what are the subgroups of $G$ satisfying above property?
I really appreciate any help you can provide.
The Artin-Wedderburn theorem says $\mathbb{C}[G]\cong\bigoplus\mathrm{End}(U)$ as $U$ ranges over all irreps of $G$.
If $N\trianglelefteq G$ is a normal subgroup then any irrep $G/N\to GL(V)$ can be turned into an irrep $G\to GL(V)$ by simply composing $G\to G/N\to GL(V)$. Then AW applies to $\mathbb{C}[G/N]$ too - the summands in $\mathbb{C}[G/N]$'s AW decomposition are precisely those in $\mathbb{C}[G]$'s decomposition corresponding to those $G$-irreps $U$ which "came from" being irreps of $G/N$ first.
Define $\rho_{G/N}$ to be this subalgebra of $\mathbb{C}[G]$. I assume by $\rho_{G/N}^\perp$ you mean the orthogonal complement with respect to the obvious inner product (in which $G$ is an orthonormal basis). Since $G$ acts on $\mathbb{C}[G]$ by left-multiplication unitarily, $\rho_{G/N}^\perp$ must be a subrep of $\rho_G=\mathbb{C}[G]$ complementary to $\rho_{G/N}$. This forces $\rho_{G/N}^\perp$ to be a direct sum of the $\mathrm{End}(U)$s corresponding to irreps $U$ that do not come from irreps of $G/N$.
The condition $(\rho_{G/M}^\perp)^H\ne0$ may be rewritten as $\dim(\rho_{G/N})^H<\dim(\rho_G)^H$.
Exercise. Show how to turn a transversal for $G/H$ into a basis for $\mathbb{C}[G]^H$.
Thus, we may further rewrite the condition as $[G:HN]<[G:H]$, or simply $N\not\subseteq H$. You can apply this to your particular situation of prime-power cyclic groups if you want.