V is a finite dimension inner product space. $T\in End(V)$ is a self adjoint operator for which $T^{2}=T$ and I'm asked to prove a subspace $U\subset V$ exists for which T is the orthogonal projection onto U.
My line of thought is that since T is self adjoint it must be invertible, and since $T^{2}=T\Rightarrow T^{-1}TT=T^{-1}T\Rightarrow T=Id_V$. But then no such subspace U exists. What is the mistake in my reasoning?
It's not true that self-adjoint implies invertible. Self-adjoint only means that $T = T^*$ (that is, $(Tx, y) = (x, Ty)$ for all $x, y \in V$.
All of the following matrices are self-adjoint with respect to the standard inner product and satisfy the equation $A^2 = A$: $$ \begin{pmatrix} I_r & 0 \\ 0 & 0 \end{pmatrix} $$ where $r = \operatorname{Rank}(A) \le n$. In fact, if $T^2 = T$ then there is a basis in which $T$ has the above form. If $T$ is self-adjoint, then there is an orthonormal basis in which $T$ has the above form.
The key property of self-adjoint operators is that if $U$ is $T$-invariant, then so is $U^\perp$. To see this, take $y \in U^\perp$. Then by definition, $(x,y) = 0$ for all $x \in U$. Now apply $T$. This gives $(x,Ty) = (Tx, y) = 0$ because $Tx \in U$. Since this works for all $x$, $Ty \in U^\perp$. So $U^\perp$ is $T$-invariant.
Now, if $T$ is the projection onto $U$ then $U = \operatorname{im}(T)$. So you need to show that $T : V \to \operatorname{im}(T)$ is an orthogonal projection.