Recently I asked about (references to) some results concerning representation theory of compact topological groups: here is the discussion
Representation theory of locally compact groups
In particular, I looked in the mentioned book of Deitmar (Prinicples of Harmonic Analysis) and I found the following result in which I was
interested in, namely that every representation of compact group splits into direct sum of irreducible ones. I don't understand everything in
the proof so let me explain the details:
1. Theorem of Peter-Weyl and orthogonality relations imply that $L^2(G)$ splits into direct (orthogonal) sum $\bigoplus_{\tau}W_{\tau}$ where
summation runs over all possible irreducible, finite dimensional (at this moment we don't know yet, that finite dimension of irreducible representation
is automatic) non-equivalent representations (we pick one such a representation from the given class) and $W_{\tau}$ is spanned by the matrix
coefficients $\tau_{ij}$ of $\tau$.
2. Let $\tau$ be a finite dimensional, irreducible representation and $\tau_{ij}$ the corresponding matrix component. From the equality
$\tau_{ij}(x)=\sum_{k}\tau_{ik}(x)\tau_{kj}(y)$ one infers that the subspaces $W_{\tau}$ are invariant under the action of right regular
representation given by $(R(y)f)(x):=f(xy)$. As I understand things correctly, we need this fact further.
Proof that every representation splits into irreducibles:
let $(\pi,V_{\pi})$ be a representation and $v,w \in V_{\pi}$. Put $\psi_{v,w}(x):=\langle \pi(x)v,w \rangle$ so $\psi:G \to \mathbb{C}$ is a
continuos function $\psi \in C(G) \subset L^2(G)$. One then computes
$$\psi_{\pi(y)v,w}(x)=\langle \pi(x)\pi(y)v,w \rangle=\langle \pi(xy)v,w \rangle=R(y)\psi_{v,w}(x).$$
This means that the mapping $T:v \mapsto \psi_{v,w}$ (where $w$ is fixed), $T: V_{\pi} \to L^2(G)$ is intertwining $\pi$ and the right regular
representation $R$. If we take $v,w$ such that $\langle v,w \rangle \neq 0$ then $Tv=\psi_{v,w} \neq 0$ (since $\psi_{v,w}(e)=\langle v,w\rangle$).
Therefore at least one component of $Tv$ in the decomposition $L^2(G)=\bigoplus_{\tau}W_{\tau}$ is nonzero: choose $P$ to be a projection onto this
component, denote this by $P:L^2(G) \to W$. Then $S:=PT$, $S:V_{\pi} \to W$ satisfies $Sv \neq 0$ in particular $S \neq 0$. One can show that $S$
intertwines $\pi$ and $R|W$ (at this moment I really use the fact that $W$ is invariant for $R$, therefore $R|W$ makes sense!)
The author claims that $V_{\pi} \cong ker S \oplus W$. I believe that we can assume, without the loss of generality that $S$ is onto: since the image
of $S$ is in $W$ which is finite dimensional then this image is closed and we can restrict everything to $im S$ (note that the image of the
intertwiner between $\pi_1,\pi_2$ is always $\pi_2$ invariant, so we can restrict the right regular representation to this image). Also we
know that $W$ is finite dimenional. But is this enough to claim that $V_{\pi} \cong kerS \oplus W$ and how this isomorphism should be understood?
We want after all, to have the decomposition into orthogonal sum! I know that the general theory of Hilbert space operators gives
$H=ker S\oplus \overline{im S^*}$ where $S:H \to K$ and $S^*:K \to H$ is the adjoint. Finally we want our pieces to be invariant so some
abstract ismorphism is not enough for us.
Proposed solution As I explained, we can assume that $S$ is onto. We take $S'$ to be $S$ restricted to the orthogonal complement of $ker S$.
Then $S'$ is injective and still it is onto: it establishes an isomorphism between $(kerS)^{\perp}$ and $W$: in particular this means that
$(kerS)^{\perp}$ is finite dimensional. Once we have this, we really don't need $W$ anymore, we only want to be able to start with
"transfinite induction" (since we know alredy-this was established earlier-that every finite dimensional representation splits into the
direct sum of irreducibles).
Is my argument correct?
Now we use Zorn's lemma, which provides us with the subspace $U$ which is maximal and has the property that $\pi|U$ splits into the direct sum of irreducibles. The author claims the orthocomplement $U^{\perp}$ has no finite dimensional subrepresentations other than zero and therefore it is zero and $U=V_{\pi}$
Why $U^{\perp}$ doe not contain any finite dimensional (nonzero) subrepresentation?
I would be grateful for any help.