I'm given that the inner product in a linear subspace $V \subset R^n$ is defined as $<X,Y>_V = X'AY$ where matrix $A$ is positive definite. I need to show that $P$ is an orthogonal projector if and only if $P^2 = P$ and $AP$ is symmetric.
Any help? This seems easy, but my grasp on the concept of projectors is minimal. Well, $P^2 = P$ is part of the definition of an orthogonal projector, but the $AP$ significance is throwing me off.
On
The projection property $P^{2}=P$ is independent of anything to do with an inner product. Orthogonality, however, is equivalent to requiring that $P$ is symmetric with respect to the given inner product. In your case, $$ (Px,y)_{A} = (x,Py)_{A} $$ is equivalent to $$ (PX)'AY=X'A(PY) \\ X'P'AY=X'APY \\ P'A = AP \\ (AP)' = AP $$ (The last step follows because $A'=A$ is assumed.) Therefore, $P$ is orthogonal with respect to the inner product $(\cdot,\cdot)_{A}$ iff $AP$ is symmetric.
Hint. A projector is said to be orthogonal if its image is orthogonal to its null space (with respect to a given inner product). Therefore, what you need to show is that $\langle Px, (I-P)y\rangle=0$ for every $x,y$ if and only if $AP$ is symmetric, under the assumption that $P^2=P$.
The forward implication is trivial. For backward implication, let $AP=S$ for some symmetric matrix $S$. To show that $P'A(I-P)=0$, prove that $A[(A^{-1}(P'A))(I-P)]=0$ instead.