Showing Normal Equations as Linear System of Equations

Question

[This is a practice problem]

I watched tutorials on least square method and normal equations and understood them too. However, i am confused with this question:

Measurement vals $p_0 = 0, p_1 = 2$, and $p_2 = 1$ were obtained at nodes $u_0 = 0, u_1 = 1$, and $u_2 = 2$. Function $p(u) = au + b$ approximates these values according to the least squares method.Formulate the normal equations as linear system.

Now I have a sample solution where to start, 2 equations are derived as followed:

$\sum p = a\sum u + 3b$ (1)

$\sum up = a\sum u^2 + b\sum u$ (2)

How are these equations derived? I couldn't find any relevant examples online. For the first one I can assume that it's 3b since there are 3 values but I don't understand the second one.

Answer 1

Multiply the linear function $p\left(u\right)=au+b$ by $u$ as shown below. $$ \begin{matrix} u_1p_1 & = & au_1^2 & + & bu_1 \\ u_2p_2 & = & au_2^2 & + & bu_2 \\ u_3p_3& = & au_3^2 & + & bu_3 \\ \end{matrix} $$

Now, add them together using the notation of summation. $$\sum_{i=1}^3 u_ip_i=a\sum_{i=1}^3 u_i^2+b\sum_{i=1}^3 u_i $$

So, you see that there is no 3.

Answer 2

Basically you minimize the squared difference between the observed values $p_i$ and the fitted value $f(u_i)$.

$$S=\sum_{i=1}^n(p_i-f(u_i))^2,$$

where $f(u_i)=a\cdot u_i+b$ and $n=3$.

$$S=\sum_{i=1}^3(p_i-a\cdot u_i-b)^2,$$

Differentiating $S$ w.r.t $a$ and $b$ and setting these partial derivatives equal to $0$.

$$\frac{\partial S}{\partial a}=\sum_{i=1}^3 2\cdot (p_i-a\cdot u_i-b)\cdot (-u_i)=0$$

$$\frac{\partial S}{\partial b}=\sum_{i=1}^3 2\cdot (p_i-a\cdot u_i-b)\cdot (-1)=0$$

Can you proceed to get the 2 equations in the form you´ve mentioned?

Answer 3

Let $$ A=\begin{bmatrix} u_1 & 1 \\ u_2 & 1 \\ u_3 & 1 \end{bmatrix} , \quad x=\begin{bmatrix} a\\ b \end{bmatrix}, \quad p =\begin{bmatrix} p_1 \\ p_2 \\ p_3 \end{bmatrix} $$ Then the least-squares solution is the vector of parameters $x$ for which $||Ax - p||_2$ is as small as possible. This solution is also the solution of the normal equations, which are given by $$ A^TA x = A^Tp.$$