Proof that trace of 'hat' matrix in linear regression is rank of X

Question

I understand that the trace of the projection matrix (also known as the "hat" matrix) X*Inv(X'X)*X' in linear regression is equal to the rank of X. How can we prove that from first principles, i.e. without simply asserting that the trace of a projection matrix always equals its rank?

I am aware of the post Proving: "The trace of an idempotent matrix equals the rank of the matrix", but need an integrated proof.

Answer 1

If $X$ is $n \times m$ with $m \le n$ and has full rank, then $rank (X) = \min(n,m) = m$, and we know $(X^T X)^{-1}$ exists.

By commutativity of the trace operator, we have

$$tr(H) := tr (X (X^T X)^{-1} X^T) = tr (X^T X (X^T X)^{-1} ) = tr[I_m] = m$$

Answer 2

Let $X$ be an $n \times r$ matrix with full rank, i.e. $\text{rank}X = r$ (this is necessary for $(X^TX)^{-1}$ to exist).

Then, using the identity $\text{tr}(AB) = \text{tr}(BA)$, we have

$\text{tr}[X(X^TX)^{-1}X^T] = \text{tr}[X^TX(X^TX)^{-1}] = \text{tr}[I_{r \times r}] = r = \text{rank}X$, as desired.

Answer 3

The hat matrix is a projection matrix so it's idempotent. i.e. it only has eigenvalues of 0 or 1.

The eigenvalues of $A^n$ are $\lambda^n$, where $\lambda$'s are the eigenvalues of $A$, which holds for any square matrix $A$.

Since H is idempotent: $H=H^2=...=H^n=...$, therefore $\lambda = \lambda^n, n=1,2,...$. And since X is full column ranked, H is full ranked, thus the number of 1 as its eigenvalues is just rank(H). Since the trace is also the sum of the eigenvalues and the sum of eigenvalues is just rank(H). Therefore the tr(H) = rank(H) = colrank(X) = rank(X). $\lambda$.