How to derive A in this expression?

Question

How can I derive A in this matrix represented equation(a closed form solution for ridge regression) $$\ a= (X^T X + αI)^ {−1} X^T y $$ (-1 here is inverse) and $a = X^TA$ I have to get $$\ α^{-1}(y-Xa) .$$ But I am not sure how can i get to it . TIA!

Answer 1

Note that $\ X^TX+αI$ and its inverse are scalars. This is because $X$ is an $n \times 1$ matrix, so $X^T$ is $1 \times n$ and thus $X^TX$ is $1 \times 1$. Thus, we can move this around without worring about its order, unlike matrices.

Use the hint as the first step (after substituting the expression for $a$): \begin{align} \ a&= (X^T X + \alpha I)^ {−1} X^T y \\ X^TA &= (X^T X + \alpha I)^ {−1} X^T y \\ (X^T X + \alpha I)X^TA &= X^T y \\ (\alpha^{-1}X^T X + I)X^TA &= \alpha^{-1}X^T y \\ \alpha^{-1}X^T X X^T A + X^T A&= \alpha^{-1}X^T y \\ X^T A&= \alpha^{-1}X^T y - \alpha^{-1}X^T X X^T A \\ (X^T)^{-1}X^T A&= \alpha^{-1}(X^T)^{-1}X^T y - \alpha^{-1}(X^T)^{-1}X^T X X^T A \\ A&= \alpha^{-1} y - \alpha^{-1} X X^T A \\ A&= \alpha^{-1} (y - X X^T A) \\ A&= \alpha^{-1} (y - X a). \end{align} Of course, this is only valid if

• $X^T X + \alpha I \neq 0$, and
• $X^T$ is invertible, which happens if and only if $X$ is invertible i.e. non-singular.