I would greatly appreciate it if someone could review and correct my solution for the following problem involving a linear equation and least squares fitting. My requirement for solutions is that I need to be able to set up equations with justification: a definition, a theorem, or an equation from Lay or Adams' books, or a logical reasoning. For example: "The columns of A span R2 because A has pivot positions in every row, see Theorem 4 in Lay, page 63." I have done my best to carefully present my steps and thoughts. Here is the problem and my solution:
- Measurements have been conducted that show a relationship between two variables, $t$ and $y$. In total, we have four data points $(-1,0),(0,1),(1,2)$, and $(2,4)$, where the first component corresponds to $t$ values and the second component to $y$ values. Here, we will attempt to find a linear equation that approximately describes the relationship between $y(t)$.
(a) Use the equation of a straight line, $k t + m = y$, along with the four data points, to set up an equation system $A x = b$.
(b) Demonstrate that there is no exact solution to $A x = b$.
(c) Formulate a least squares problem and find the least squares solution to $A x = b$ .
(d) Calculate the differences in $y$ between what is measured and what is described by the least squares fit.
We want to find a linear equation that describes the relationship between $y(t)$. First, we want to use the equation of a straight line, $k t + m = y$, along with the four given data points $(-1,0),(0,1),(1,2)$, and $(2,4)$, to set up an equation system $A\vec{x} =\vec{b}$. This equation system has solutions (is consistent) only if $\vec{b}$ lies in Col A. Since the number of equations is greater than the number of unknowns, it is possible that our linear system is overdetermined but also inconsistent. The equation system looks as follows:
$$A\vec{x} =\vec{b} \Rightarrow \left[ \begin{array}{cc}-1&1\\ 0&1\\ 1&1\\ 2&1\end{array} \right] \begin{bmatrix}k\\ m\end{bmatrix} =\begin{bmatrix}0\\ 1\\ 2\\ 4\end{bmatrix}$$ Which is the same as $$k\begin{pmatrix}-1\\ 0\\ 1\\ 2\end{pmatrix} +m\begin{pmatrix}1\\ 1\\ 1\\ 1\end{pmatrix} =\begin{pmatrix}0\\ 1\\ 2\\ 4\end{pmatrix}$$
The vector $\vec{b}$ needs to be a linear combination of the column vectors in A, and if there is a solution, we say that $\vec{b}$ is in the column space of A. Since we only have two linearly independent vectors in the matrix A, we cannot span the four-dimensional space. From matrix A, it is clear that it is an $m \times n$ matrix, where m is 4 and n is 2, and because the number of rows is greater than the number of columns, it is usually impossible to obtain any solutions, and the system is said to be inconsistent. Furthermore, the system is overdetermined since there are more equations than unknown variables. By using the data points, I could find the corresponding equation system:
[ \begin{cases} -k + m &= 0 \\ m &= 1 \\ k + m &= 2 \\ 2k + m &= 4 \end{cases} ]
From this, I obtained the solution $k = \frac{3}{2},\ m = 1$, by using the second equation along with the fourth equation. However, a solution is missing as, according to equation 1, we should have $k = m$, or $k+m=-2$.
(b) In part (a), I have shown that there is no exact solution to the equation system $A x = b$ since the system is inconsistent, and we get different solutions that are not exactly unique.
(c) To solve the problem with least squares fitting, we used a matrix form $A x = b$, where $A$ contains the coefficients for $k$ and $m$, $x$ is a vector with the unknown variables $k$ and $m$, and $b$ is a vector with the measured $y$ values. Since there is no exact solution, we use the least squares method to find the best-fitted values for $k$ and $m$.
\begin{aligned} & A^T A x=A^T b \\ & {\left[\begin{array}{cccc} -1 & 0 & 1 & 2 \\ 1 & 1 & 1 & 1 \end{array}\right]\left[\begin{array}{cc} -1 & 1 \\ 0 & 1 \\ 1 & 1 \\ 2 & 1 \end{array}\right]\left[\begin{array}{c} k \\ m \end{array}\right]=\left[\begin{array}{cccc} -1 & 0 & 1 & 2 \\ 1 & 1 & 1 & 1 \end{array}\right]\left[\begin{array}{l} 0 \\ 1 \\ 2 \\ 4 \end{array}\right]} \\ & {\left[\begin{array}{cc} (-1)(-1)+1+4 & (-1)(1)+1+2 \\ -1+1+2 & 4 \end{array}\right]\left[\begin{array}{c} k \\ m \end{array}\right]=\left[\begin{array}{c} 10 \\ 7 \end{array}\right]} \\ & \Leftrightarrow\left[\begin{array}{ll} 6 & 2 \\ 2 & 4 \end{array}\right]\left[\begin{array}{l} k \\ m \end{array}\right]=\left[\begin{array}{c} 10 \\ 7 \end{array}\right] \Rightarrow\left[\begin{array}{l} k \\ m \end{array}\right]=\left[\begin{array}{ccc} 1 & 0 & \frac{13}{10} \\ 0 & 1 & \frac{11}{10} \end{array}\right] \\ & \end{aligned}
(d) Finally, we calculate the differences in $y$ between the actual measured values and the values obtained from the least squares fitting to assess the quality of the fit. We had the data points $(-1, 0), (0, 1), (1, 2), (2, 4)$ in the form (t, y). We plug our y-values into the equation of the straight line to examine the deviation, $$y_1 = 1.3(-1) + 1.1 \Longleftrightarrow y_1 = -0.2$$ $$y_2 = 1.3(0) + 1.1 \Longleftrightarrow y_2 = 1.1$$ $$y_3 = 1.3(1) + 2 \Longleftrightarrow y_3 = 3.3$$ $$y_4 = 1.3(2) + 4 \Longleftrightarrow y_4 = 6.6$$