Solution to linear regression lies in the space of training examples - why?

Question

In the slides at https://www.cs.cmu.edu/~tom/10701_sp11/slides/Kernels_SVM_04_7_2011-ann.pdf, the author claims that w lies in the space spanned by the rows of X (see image). I don't see why this is true. Is it obvious? For this to be true, w should be expressible as $w=X^T a$.

Answer 1

Here is how I am able to prove it. Let $X = UDV^T$.

$$w = (X^TX)^{-1}X^Tb = VD^{-1}U^T b$$ Now, the columns of V lie in the row-space of X. Hence w lies in the row-space of X.

Answer 2

You were right, my previous answer was void. I am sorry about that.

Here is a correct proof:

Firstly remark that $(X^TX+\lambda I)^{-1}X^T=X^T(XX^T+\lambda I)^{-1}$,

secondly let $z:=(XX^T+\lambda I)^{-1}Y$

Conclude that $w=X^Tz$.