Showing Orthogonal Projection Matrix Multiplied by Full-Rank Matrices is Positive-Definite

Question

Here is some useful information pertaining to my question:

• Let $X \in\mathbb{R}^{n \times m}$ and $Z \in\mathbb{R}^{n \times p}$ be full-rank matrices.
• Define $B = I_n - X(X^{T}X)^{-1}X^{T}$.
• Assume that the columns of $X$ are linearly independent from the columns of $Z$.

I am trying to show that $Z^{T}BZ$ is positive definite.

My first plan of attack was to (1) show that $Z^{T}BZ$ is an orthogonal projection matrix, (2) prove that $I$ is the only positive definite orthogonal projection matrix, and (3) prove that $Z^{T}BZ=I$. This fell through because I felt that there was not enough information about $Z$ to prove (1).

I am pretty sure that I will ultimately need to show either (i) $z^{T}Z^{T}BZz>0$ $\forall$ $z\in \mathbb{R}^p$ OR (ii) all the eigenvalues of $Z^{T}BZ$ are positive. I think my biggest problem stems from not knowing how to deal with $Z^{T}$ and $Z$... clearly they are important, as it seems that we can only prove that $B$ is positive semidefinite. However, I am unsure how this full-rank matrix can "transform" $B$ from psd to pd.

Any assistance would be greatly appreciated! If it helps, I have already shown that B is an orthogonal projection matrix and that B is psd. :)

Answer 1

Hmm, I am a bit surprised that no one went and finished the proof.

So how do we prove that $ ( I - X ( X^T X )^{-1} X^T ) Z $ has linearly independent columns? Assume that it doesn't and show that this results in a contradiction. We will look at $ Z - X ( X^T X )^{-1} X^T Z $ instead.

Let $ x $ be such that $ (Z - X ( X^T X )^{-1} X^T Z) x = 0 $

• Maybe $ Z x = 0 $. But that can't be because $ Z $ has linearly independent columns. So, $ Z x \neq 0 $.

• Maybe $ X^T ( Z x ) = 0 $. That can't be since then $ Z x - X ( X^T X )^{-1} X^T Z x = Z x = 0 $ because $ Z $ has linearly independent columns. So, $ X^T Z x \neq 0 $. This also means $ y = ( X^T X )^{-1} X^T Z x \neq 0 $.

This reasoning tells us that $ Z x - X y = 0 $ or, equivalently, $ Z x = X y $ for both $ x \neq 0 $ and $ y \neq 0 $. But that in turn means that the columns of $ Z $ are not linearly independent of the columns of $ X $ since then $$ \left( \begin{array}{c | c} Z & X \end{array} \right) \left( \begin{array}{c} x\\-y \end{array} \right) = 0. $$

Answer 2

This is an interesting question.

Some observations:

• $ I - X ( X^T X)^{-1} X^T $ projects onto the subspace orthogonal to the column space of $ X $.

• Applying a projection twice vs. once does not change the result.

• A projection is symmetric.

So, $Z^T B Z = Z^T B B Z = Z^T B^T B Z = ( B Z )^T B Z $.

Thus, the equivalent question is to prove that $ B Z $ has linearly independent columns. ($ A^T A $ is positive definite if and only if $ A $ has linearly independent columns.)

Intuitively, $ B Z $ should have linearly independent columns since each column of $ Z$ needs to have a component orthogonal to the column space of $ X $ or else it is in the column space of $ X $.

I'll let someone else formalize the last observation.

Answer 3

To prove positive definite, we must show that $x^TZ^TBZx=0$ implies $x=0$. We have $B^TB=B$, since $B$ is symmetric and idempotent. Hence $x^TZ^TBZx=(BZx)^T(BZx)=\|BZx\|^2$ and therefore $BZx=0$.

Substituting $B:=I-X(X^TX)^{-1}X^T$, this means $Zx-Xy=0$, where $y:=(X^TX)^{-1}X^TZx.$ Since the columns of $Z$ are linearly independent of the columns of $X$, this implies that $Zx=0$ or $Xy=0$.

• If $Zx=0$, this implies $x=0$ since the columns of $Z$ are linearly independent.
• If $Xy=0$, then $Zx=Zx-Xy=0$ so again $x=0$.