Using the method of least squares in order to find the best approximation to a solution for the system

Question

Use the method of least squares in order to find the best approximation to a solution for the system $$3x + y = 1\\ x − y = 2\\ x + 3y = −1$$

My Try: $$Ax=B$$ $$\begin{bmatrix} 1 & 1 \\ 1 & -1 \\ 1 & 3 \end{bmatrix}\begin{bmatrix} x \\ y \end{bmatrix}=\begin{bmatrix} 1 \\ 2 \\ -1 \end{bmatrix}$$

First I found $A^TA$ and then $A^TB$

$$A^TA=\begin{bmatrix} 1 & 1 & 1 \\ 1 & -1 & 3 \end{bmatrix}\begin{bmatrix} 1 & 1 \\ 1 & -1 \\ 1 & 3 \end{bmatrix}=\begin{bmatrix} 3 & 3 \\ 3 & 11 \end{bmatrix}$$ and $$A^TB=\begin{bmatrix} 1 & 1 & 1 \\ 1 & -1 & 3 \end{bmatrix}\begin{bmatrix} 1 \\ 2 \\ -1 \end{bmatrix}=\begin{bmatrix} 2 \\ -4 \end{bmatrix}$$ The I used gaussian elimination for $\begin{bmatrix} 3 & 3 & 2 \\ 3 & 11 & -4 \end{bmatrix}$ and got $x=\dfrac{17}{12},y=-\dfrac{3}{4}$.

Is this method of least squares correct?

Answer 1

Yes, the method is correct. It is more common to write it as: $$ x = (A^TA)^{-1}A^TB$$

Your calculations are right, except that you have a typo in the matrix $A$, as pointed out in the comments.

Answer 2

Since you did not specify that matrix calculation was a requirement, admitting that you need to solve $n$ linear equations written as $$a_ix+b_iy=c_i \qquad \qquad (i=1,2,\cdots,n)$$ with $n>2$, by analogy with least squares method, you could simply minimize with respect to $(x,y)$ the norm $$\Phi=\frac 12\sum_{i=1}^n(a_ix+b_iy-c_i)^2$$ Computing the partial derivatives $$\frac{\partial \Phi}{\partial x}=\sum_{i=1}^na_i(a_ix+b_iy-c_i)$$ $$\frac{\partial \Phi}{\partial y}=\sum_{i=1}^nb_i(a_ix+b_iy-c_i)$$ Setting them equal to $0$, you then end with two linear equations in $(x,y)$, namely $$x \sum_{i=1}^n a_i^2+y \sum_{i=1}^n a_ib_i=\sum_{i=1}^n a_ic_i$$ $$x \sum_{i=1}^n a_ib_i+y \sum_{i=1}^n b_i^2=\sum_{i=1}^n b_ic_i$$