solution of regression coefficient using normal equations

Question

I have a really stupid question. We know the solution of regression coefficient is the following $$ \hat\beta = (X^tX)^{-1}X^tY $$

Can I further expand this into the following ? $$ \hat\beta = (X^tX)^{-1}X^tY = X^{-1}(X^t)^{-1}X^tY = X^{-1}IY = X^{-1}Y$$

What is wrong in this?

Answer 1

Note that usually when dealing with regression $X$ is not a square matrix thus we can’t simplify the expression .

Answer 2

For simplicity, consider $y=2x$: $$\begin{array}{c|lcr} x & y \\ \hline 1 & 2 \\ 2 & 4 \\ 3 & 6 \end{array}$$ Calculate the coefficient: $$\begin{align}\hat\beta = (X^tX)^{-1}X^tY &=\left(\pmatrix{1&2&3}\cdot \pmatrix{1\\2\\3} \right)^{-1}\cdot \pmatrix{1&2&3}\cdot \pmatrix{2\\4\\6}=\\ &=(14)^{-1}\cdot \pmatrix{28}=2.\end{align}$$ And now try to calculate the elements of your rearranged formula. Can you find $X^{-1}$? Do you see the issue?

Answer 3

Typically $X$ has many more rows than columns, so for example we might have $$ \begin{bmatrix} 1 & 1 \\ 1 & 2 \\ 1 & 3 \\ 1 & 4 \\ 1 & 5 \\ 1 & 6 \end{bmatrix}. $$ In this case $X^TX$ is a $2\times 2$ matrix. It is invertible because the columns of $X$ are linearly independent. But what would one use as the inverse of $X$ if one writes $X^{-1}(X^T)^{-1}\text{ ?}$

A matrix with linearly independent columns has a left inverse, but has no two-sided inverse unless it has as many columns as rows.