Least Squares Approximation

Question

Given:

$\begin{bmatrix} 1 & -2\\ -2 & 4 \end{bmatrix} \cdot \begin{bmatrix} x\\ y \end{bmatrix} = \begin{bmatrix} 1\\ 5 \end{bmatrix}$

How do I find the least squares approximation and compute the error.

I understand I should use:

$\hat{x} = (A^TA)^{-1}A^Ty$

Where $y = \begin{bmatrix} 1\\ 5 \end{bmatrix}$

However, looking at $A = \begin{bmatrix} 1 & -2\\ -2 & 4 \end{bmatrix}$ and applying to $(A^TA)^{-1}$ I dont understand how to finish the projection because it doesnt have an inverse.

Could someone maybe explain it step-by-step so I can wrap my head around it? I understand that I am trying to find the vector that is on the plane closest to the vector not on the plane, but how do I compute the error for it? Isint least squares already the smallest error?

Thanks

Answer 1

Instead of finding the inverse, solve $Mx=b$ using Gaussian elimination, where $M=(A^TA)$ and $b=A^Ty$.

Answer 2

step one: differentiate $(Ax-b)^2$ get $A^TAx=A^Tb$
$\qquad$ This means for any x s.t. $A^TAx=A^Tb$, it will give you a maximum or minimum. Therefore there are three cases: 1. the result exists and is unique, 2. the result exists and is not unique, 3. the result does not exist. (Just solve two simutaneous equations.)

step two: Since $(Ax-b)^2$ can be any large, the result must be minimum. The solution is: $x^*$=[-0.5556 1.1111]. Error = $(Ax^*-b)^2$