My question is about "proving that $\mathbf{I} - \mathbf{X} ( \mathbf{X}' \mathbf{X} )^{-1} \mathbf{ X }' $ is positive semidefinite?"
For instance, I attempted proving that $$\mathbf{c'} [\mathbf{I} - \mathbf{X} ( \mathbf{X}' \mathbf{X} )^{-1} \mathbf{ X }']\mathbf{c} \ge 0$$
where $\mathbf{c}$ is any n-dimensional non-zero vector, $\mathbf{I}$ is an n-dimensional identity matrix and $\mathbf{X}$ is a matrix of order n x k. Now, taking the LHS of the inequality above,
$$ \mathbf{c'c} - \mathbf{c'}\mathbf{X} ( \mathbf{X}' \mathbf{X} )^{-1} \mathbf{ X }'\mathbf{c}$$
Letting $\mathbf{z}$ = $\mathbf{ X }'\mathbf{c}$, I am able to conclude that [$\mathbf{z'}( \mathbf{X}' \mathbf{X} )^{-1}\mathbf{z}]\ge0$, since $( \mathbf{X}' \mathbf{X} )^{-1}$ is positive semi-definite.
Also, I know that $\mathbf{c'c}>0$ since it is the sum of squares of elements in $\mathbf{c}$.
But, I am stuck here regarding how to prove $\mathbf{c'c} \ge [\mathbf{z'}( \mathbf{X}' \mathbf{X} )^{-1}\mathbf{z}]$.
Kindly help...
$\newcommand{\x}{\mathbf{x}}\newcommand{\A}{\mathbf{A} }$Hints: Let $\A = \mathbf{I} - \mathbf{X} ( \mathbf{X}' \mathbf{X} )^{-1} \mathbf{ X }' $. Then you can show that $\A^2 = \A$. This implies that the eigenvalues of $\A$ all satisfy $\lambda^2=\lambda$, that is, the eigenvalues are all $0$ or $1$. Furthermore, it is easy to show that $\A$ is symmetric.
Thus $\mathbf{A}$ is a real symmetric matrix with only non-negative eigenvalues, which implies that it is positive semi-definite.
Alternatively: once you have shown that $\A^2 = \A$ and $\A'=\A$, note that $\x' \A \x = \x' \A^2\x = \x'\A'\A\x = (\A\x)'(\A\x) = \left\|\A\x\right\|^2$ for all $\x \in \Bbb{R}^n$.