Hi everyone I have the following question and I hope you can help me.
Let us suppose that $G$ is a reductive group and $H$ a normal subgroup. In this situation, $H$ and $G/H$ are reductive and therefore there exists Reynold's operator for $G$, $H$ and $G/H$ which we will denote by $R_{G}$, $R_{H}$ and $R_{G/H}$. If we have a $G$-module (this is a vector space $V$ together with an action of $G$ in $V$), then $V$ is a $H$-module through the canonical inclusion $H\hookrightarrow G$ and $R_{H}(V)=V^{H}$ is a $G/H$-module because $H$ acts trivially on $V^{H}$.
It is true that $R_{G}=R_{G/H}\circ R_{H}$? How can I prove it?
Thank you for your time.
A vector $v \in V$ is invariant for $G$ iff it is invariant for $H$ and is invariant for the induced action of $G/H$ on $V^H$. This is true for any normal subgroup $H$ of a group $G$ acting on any set $X$.
$R_H$ is the projection onto the $H$-invariants with the kernel the sum of all irreducible non-trivial $H$-subrepresentations. Similarly $R_{G/H}$ is the projection of $V^H$ to the $G/H$-invariants, with kernel the sum of all irreducible non-trivial $G/H$-subrepresentations in $V^H$.
Hence their composition is a projection onto the $G$-invariants of $V$, by the observation in the first paragraph. So we need to check that the kernel of this composition is the sum of all non-trivial $G$-subrepresentations of $V$.
Note that since $H$ is normal in $G$ every irreducible representation of $G$ is either trivial as an $H$-representation or has no $H$-invariant vectors. In particular the kernel of $R_H$ is also the direct sum of all $G$-subrepresentations of $V$ that are not trivial on $H$. So the total kernel of this composition is: all irreducible $G$-subreps that are not $H$-invariant (i.e. $ker(R_H)$) direct sum all irreducible $G$-subreps that are $H$-invariant but not invariant for all of $G$ (i.e. $ker(R_{G/H})$). Clearly though this is just the sum of all non-trivial $G$-subrepresentations of $V$, as desired.