I have three points $(-1,7)$, $(1,7)$ and $(2,21)$ and a linear model $b = C + Dt$. It is asking to give and solve the normal equations in $\hat{x}$ to find the projection $p = A\hat{x}$ of $b$ onto the column space of $A$ (i.e. find $\hat{x}$ with least $||A\hat{x} - b||^2$).
I think I've found the normal equations to be $3C + 2D = 35$ and $2C + 6D = 42$ and the $b = 9 + 4t$, but I'm not sure where to go from there.
Problem statement
Given a sequence of $m=3$ data points of the form $\left\{ x_{k}, y_{k} \right\}$, and a model function $$ y(x) = a_{0} + a_{1} x $$ find the least squares solution $$ a_{LS} = \left\{ a\in\mathbb{C}^{n} \colon \lVert \mathbf{A} a - y \rVert_{2}^{2} \text{ is minimized} \right\} $$
Linear system
$$ % \begin{align} % \mathbf{A} a & = y \\ % \left[ \begin{array}{cr} 1 & -1 \\ 1 & 1 \\ 1 & 2 \\ \end{array} \right] % \left[ \begin{array}{c} a_{0} \\ a_{1} \end{array} \right] % &= \left[ \begin{array}{c} 7 \\ 7 \\ 21 \\ \end{array} \right] \end{align} % $$
Normal equations
$$ % \begin{align} % \mathbf{A}^{*} \mathbf{A} a &= \mathbf{A}^{*} y \\ % \left[ \begin{array}{cc} 3 & 2 \\ 2 & 6 \\ \end{array} \right] % \left[ \begin{array}{c} a_{0} \\ a_{1} \end{array} \right] % &= % \left[ \begin{array}{c} 35 \\ 42 \end{array} \right] % \end{align} % $$
Solution via normal equations
$$ % \begin{align} % a_{LS} &= \left( \mathbf{A}^{*}\mathbf{A} \right)^{-1} \mathbf{A}^{*}y \\ % &=\frac{1}{14} % \left[ \begin{array}{rr} 6 & -2 \\ -2 & 3 \\ \end{array} \right] % \left[ \begin{array}{c} 35 \\ 42 \end{array} \right] \\ % &= % \left[ \begin{array}{c} 9 \\ 4 \end{array} \right] % \end{align} % $$
Solution function
$$ y =9 + 4 x $$
Is the best fit a good fit?
The residual error vector is $$ r =\mathbf{A} a_{LS} - y = \left[ \begin{array}{r} -2 \\ 6 \\ -4 \end{array} \right] $$
The function which was minimized is the total error $$ r^{2} = r \cdot r = 56. $$
Solution plotted against the data points:
Projections
What is the projection on the column space?
$$ \mathbf{A} a = a_{0} \mathbf{A}_{1} + a_{1} \mathbf{A}_{2} = 9 \left[ \begin{array}{r} 1 \\ 1 \\ 1 \end{array} \right] + 4 \left[ \begin{array}{r} -1 \\ 1 \\ 2 \end{array} \right] = % \left[ \begin{array}{r} 5 \\ 13 \\ 17 \end{array} \right] \in \mathcal{R}\left( \mathbf{A} \right) $$
Errata
Given an invertible matrix $\mathbf{A}$, the inverse matrix can be computed using
$$ \mathbf{A}^{-1} = \frac{\text{adj }\mathbf{A}} {\det \mathbf{A}} $$ where the adjugate matrix adj $\mathbf{A}$ is the matrix of cofactors and $\det \mathbf{A}$ is the determinant.
For the product matrix in the example, the adjugate matrix is $$ \text{adj }\mathbf{A}^{*}\mathbf{A} = % \left[ \begin{array}{rr} 6 & -2 \\ -2 & 3 \\ \end{array} \right] % $$ and the determinant is $$ \det \mathbf{A}^{*}\mathbf{A} = 18 - 4 = 14. $$ The inverse of the product matrix is $$ \left( \mathbf{A}^{*}\mathbf{A} \right) ^{-1} = \frac{1}{14} % \left[ \begin{array}{rr} 6 & -2 \\ -2 & 3 \\ \end{array} \right]. $$ Thanks to @spacedustpi for clarification.