NOTE: The beginning part, which might sound like a lot of complicated theory to some, is just for completeness. I might just need $\operatorname{dim}V^G = \operatorname{dim}(V^\ast)^G$, which anyone who knows what a representation is should understand. It is also very likely that one can completely ignore the property "rational".
In Computational Invariant Theory by Harm Derksen and Gregor Kemper on page 39-40, theorem 2.2.5 states that if a linear algebraic group $G$ is linearly reductive, then for every rational representation $V$ of $G$ there exists a unique subrepresentation $W$ of $V$ such that $V = V^G \oplus W$ (i.e. $W$ is the unique module complement of $V^G$). This is the main statement I would like to understand.
A linear algebraic group $G$ is called linearly reductive if and only if for every rational representation $V$ we have the following: For all $v \in V^G \setminus \{0\}$ there exists a linear invariant $\varphi \in (V^\ast)^G$ such that $\varphi (v) \neq 0$. (See Computational Invariant Theory, p.38, definition 2.2.1)
In other words, $G$ is linearly reductive if and only if for every rational representation $V$ of $G$, $\left. b \right|_{(V^\ast)^G \times V^G}$ is non-degenerate in the right variable, where $b \colon V^\ast \times V \rightarrow K$, $(\varphi,v) \mapsto \varphi(v)$ is the canonical pairing.
To show the statement, the book claims that for a linearly reductive group $G$ and a rational representation $V$, we have that $V^G$ and $(V^\ast)^G$ are dual to each other, which is equivalent to saying that $\left. b \right|_{(V^\ast)^G \times V^G}$ is non-degenerate in both variables. This is what I cannot show and what I am having difficulties with.
We have non-degeneracy in the right variable by definition of "linearly reductive", so it suffices (it is in fact equivalent) to show that $\operatorname{dim}V^G = \operatorname{dim}(V^\ast)^G$. Again, I just can't prove this, I always fail.
Notes:
- I am pretty sure that "rational" isn't needed, the statement should also be true if we replace " rational representation" by "representation", and replace "linear algebraic group" by "group".
- I believe that $\operatorname{dim}V^G = \operatorname{dim}(V^\ast)^G$ is true for any representation, even if $G$ isn't linearly reductive, but I am not sure. In any case, if we can show this, it would be all I need in order to show everything I need.
Some other definitions that some might not know or have forgotten:
- If $G$ acts on $V$, then $G $acts on $V^\ast$ via $\sigma.\varphi(v) := \varphi(\sigma^{-1}.v)$, where $\sigma \in G$, $\varphi \in V^\ast$, $v \in V$. $V^\ast$ is a representation of $G$ via this definition.
- If $V$ is a representation of a group $G$, then $V^G := \{\, v \in V \mid \forall \sigma \in G : \sigma.v = v \,\}$ is the space of all invariants. This is a subrepresentation of $V$.
Can anyone help me? Thanks.
EDITS:
- "representation" here means "finite representation"
Since $V$ is finite dimensional, the canonical map $V \to (V^*)^*$ is an isomorphism of $G$-representations. Consider the map $$V \cong (V^*)^* \to ((V^*)^G)^* \quad \text{given by restriction} \quad v \mapsto v|_{(V^*)^G}.$$ This map is surjective with kernel $K$ equal to the space of all $v \in V$ such that $\phi(v)=0$ for all $\phi \in (V^*)^G$. The group $G$ acts trivially on the target $((V^*)^G)^*$ of this map, and if $U \subseteq V$ is a subspace such that $G$ acts trivially on $V/U$, then each linear functional on $V/U$ defines an element of $(V^*)^G$. It follows that $K \subseteq U$, and hence $((V^*)^G)^*$ is the largest quotient of $V$ on which $G$ acts trivially.
This quotient is usually called the space of co-invariants of $V$, and is typically denoted $V_G$. By construction it is dual to the space $(V^*)^G$ you are interested in, and replacing $V$ by $V^*$ we see that $(V^*)_G$ is dual to $V^G$. On the other hand, there is a natural (functorial in $V$) map $V^G \to V_G$ obtained by composing the inclusion $V^G \subseteq V$ with the projection $V \to ((V^*)^G)^*=V_G$.
The condition that for each $v \in V^G \setminus \{0 \}$ there is some $\phi \in (V^*)^G$ with $\phi(v) \neq 0$ implies that the intersection of $K$ with $V^G$ is $0$, or in other words that the canonical map $V^G \to V_G$ is injective for all representations $V$. Applying this to $V^*$ shows that the canonical map $(V^*)^G \to (V^*)_G$ is injective as well. Thus $$\mathrm{dim}(V_G)=\mathrm{dim}(V^*)^G \leq \mathrm{dim}(V^*)_G=\mathrm{dim}(V^G).$$ Hence the map $V^G \to V_G=((V^*)^G)^*$ is an isomorphism for all representations, so $V^G$ and $(V^*)^G$ are dual to one another as claimed.
To see that the reductive condition is actually necessary, you should look at the prototypical non-reductive groups: groups of upper triangular unipotent matrices. So let $k$ be a field and consider the group $U$ of linear transformations of $V=k^3$ consisting of all matrices $u$ of the form $$u=\left( \begin{matrix} 1 & a & b \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{matrix} \right).$$ One checks that $V^U$ is the span of the first basis vector, hence of dimension $1$, and also that the quotient by it is the space $V_U$ of co-invariants, which is therefore of dimension $2$.