I have this quote from the book Mathematical Methods for Physics and Engineering, page 1188:
"The set of $g$ matrices that forms an $n$-dimensional representation $D$ of the group $G$ can be thought of as acting on column matrices corresponding to vectors in an $n$-dimensional vector space $V$ spanned by the basis functions of the representation. If there exists a proper subspace $W$ of $V$, such that if a vector whose column matrix is $w$ belongs to $W$ then the vector whose column matrix is $D(X)w$ also belongs to $W$, for all $X$ belonging to $G$, then it follows that $D$ is reducible. We say that the subspace $W$ is invariant under the actions of the elements of $G$. With $D$ unitary, the orthogonal complement $W_⊥$ of $W$, i.e. the vector space $V$ remaining when the subspace $W$ has been removed, is also invariant[.]"
I don't get the last part: why does $V \setminus W$ orthogonal to $W_⊥$?
I do not get this because why cannot $V \setminus W$ contain vector subspaces which are neither parallel or orthogonal to $W_⊥$? If they are parallel to $W_⊥$, they are part of subvectorspace $W_⊥$, not $V \setminus W$. If they are orthogonal, they are invariant according to the quote. But what happens to those subspaces which are neither parallel or orthogonal? (There is no subvectorspace according to the quote, if I interpret well, but why?)
I think the answer has to do with $D$ being unitary, but to me that means that it represents a rotation in the vector space without scaling, and cannot relate this to the problem above.
The highlighted passage is clumsy. Here is what is meant:
The vector space $V$ is an inner product space, with inner product $(\cdot,\cdot)$. Assume that the representation $D:G\to GL(V)$ is a unitary representation, so $$(D(g)v_1,D(g)v_2)=(v_1,v_2)$$ for all $v_1,v_2\in V$ and $g\in G$. Further, assume that $W$ is an invariant subspace under the action of $G$ (i.e. $D(g)w\in W$ for all $w\in W$). Then, the orthogonal compliment $$W_\perp=\{v\in V\mid (v,w)=0\mbox{ for all }w\in W\}$$ is also invariant under the action of $G$. Indeed, suppose that $w'\in W_\perp$, $w\in W$ and $g\in G$. Then, $$ (D(g)w',w)=(D(g^{-1})D(g)w',D(g^{-1})w)=(w',D(g^{-1})w)=0 $$ since $D(g^{-1})w\in W$ and $w'\in W_\perp$. It now follows that $D(g)w'\in W_\perp$. Since $g\in G$ and $w'\in W_\perp$ are arbitrary, it follows that $W_\perp$ is $G$-invariant.