Proof that combined matrices are idempotent.

Question

Suppose that $X_{nxp} (n>p)$ is a matrix such that $X'X$ is invertible.

• Prove that both $X(X'X)^{-1}X'$ and $I_n-X(X^TX)^{-1}X^T$ are idempotent.
• Prove that the tr $X(X'X)^{-1}X' =p$ and tr $(I_n - X(X'X)^{-1}X')=n-p$.

Answer 1

We have

$(X(X'X)^{-1}X')^2 = (X(X'X)^{-1}X')(X(X'X)^{-1}X') = X(X'X)^{-1}(X'X)(X'X)^{-1}X'=X(X'X)^{-1}X', \tag 1$

which shows that $X(X'X)^{-1}X'$ is idempotent. Now note that for any idempotent $P$,

$P^2 = P, \tag 2$

we have

$(I - P)^2 = I - 2P + P^2 = I - 2P + P = I - P \tag 3$

is also idempotent; applying this fact to $X(X'X)^{-1}X'$ shows that $I_n - X(X'X)^{-1}X'$ is idempotent as well.

We compute the trace of $X(X'X)^{-1}X'$ as follows: in the wikipedia article on traces, it is shown that for $A$ and $n \times m$ matrix and $B$ and $m \times n$ matrix,

$\text{tr}(AB) = \text{tr}(BA); \tag 4$

applying this to $X(X'X)^{-1}$ and $X'$ yields

$\text{tr}((X(X'X)^{-1})X') = \text{tr}(X'(X(X'X)^{-1}))$ $= \text{tr}((X'X)(X'X)^{-1}) = \text{tr}(I_p) = p, \tag 5$

where $I_p$ is the $p \times p$ identity matrix. Then

$\text{tr}(I_n - X(X'X)^{-1})X') = \text{tr}(I_n) - \text{tr}(I_p) = n - p. \tag 6$

Answer 2

Let us put $$ A \colon= X \left(X^\prime X\right)^{-1} X^\prime. $$ Then we note that $$ \begin{align} A^2 &= AA \\ &= \left[ X \left(X^\prime X\right)^{-1} X^\prime \right] \left[ X \left(X^\prime X\right)^{-1} X^\prime \right] \\ &= X \left[ \left(X^\prime X\right)^{-1} \left( X^\prime X \right) \right] \left(X^\prime X\right)^{-1} X^\prime \\ &= X I_{p\times p} \left(X^\prime X\right)^{-1} X^\prime \\ &= X \left(X^\prime X\right)^{-1} X^\prime \\ &= A. \end{align} $$

Now let us put $$ B \colon= I_n-X(X^TX)^{-1}X^T. $$ Then we find that $$ \begin{align} B^2 &= BB \\ &= \left[ I_n-X(X^TX)^{-1}X^T \right] \left[ I_n-X(X^TX)^{-1}X^T \right] \\ &= I_n \left[ I_n- X(X^TX)^{-1}X^T \right] - X(X^TX)^{-1}X^T \left[ I_n-X(X^TX)^{-1}X^T \right] \\ &= I_n I_n - I_n X(X^TX)^{-1}X^T - X(X^TX)^{-1}X^T I_n + \left[ X(X^TX)^{-1}X^T \right] \left[ X(X^TX)^{-1}X^T \right] \\ &= I_n - X(X^TX)^{-1}X^T - X(X^TX)^{-1}X^T + X\left[ (X^TX)^{-1} \left( X^T X\right) \right] \left[ (X^TX)^{-1}X^T \right] \\ &= I_n - 2 X(X^TX)^{-1}X^T + X I_n \left[ (X^TX)^{-1} X^T \right] \\ &= I_n - 2 X(X^TX)^{-1}X^T + X \left[ (X^TX)^{-1} X^T \right] \\ &= I_n - 2 X(X^TX)^{-1}X^T + X (X^TX)^{-1} X^T \\ &= I_n - X(X^TX)^{-1}X^T \\ &= B. \end{align} $$