Least squares fitting using cosine function?

Question

Hello I am trying to fit a harmonic of the form $$y = b + c\cos(x)$$ to four data points (0,6.1) (.5,5.4) (1,3.9) (1.5,1.6) using least squares for homework. I know that the error $= Y_i - f(x_i)$ but am pretty confused with the partial derivatives and linear algebra part of this question. Apparently the professor wants only $y=b+c\cos(x)$ and not $y=b+c\cos(x)+d\sin(x)$

I get two equations from taking the derivative of $E = \sum_{i=0}^n [y_i - (b+c\cos(x_i))]^2$, with respects to $b$ and $c$.

$$ \frac{dE}{db} = bn + \sum_{i=0}^n c\cos(x_i) = \sum_{i=0}^n y_i $$ and $$ \frac{dE}{dc} = b\cos(x_i) + \sum_{i=0}^n c\cos^2(x_i) = \sum_{i=0}^n \cos(x_i) y_i $$

Putting this into matrix form gives me: $$ \left[ \begin{array}{cc|c} n & \sum_{i=0}^n (x_i) & \sum_{i=0}^n (y_i) \\ \sum_{i=0}^n \cos(x_i) & \sum_{i=0}^n \cos^2(x_i) & \sum_{i=0}^n \cos(x_i)y_i \\ \end{array}\right] $$

which lands me with $$ \left[\begin{array}{cc|c} 4 & 3 & 17 \\ 2.48862 & 2.067077 & 13.05931 \end{array}\right] $$ Sorry for the terrible formatting | is meant to make this an augmented matrix (the four data points are included at the top). Anyways I'm not really sure how to solve this matrix so I assume I did something wrong with the partial derivatives. Is there anyone out there that could be so helpful as to let me know what I did wrong or any hints on how to solve this system if it is indeed solvable?

Answer 1

Differentiating $E = \sum_{i=0}^n [y_i - (b+c\cos(x_i))]^2$ I get $$ \frac{\partial E}{\partial b} = -2 \sum_{i=0}^n [y_i - (b+c\cos(x_i))] $$ and $$ \frac{\partial E}{\partial c} = -2 \sum_{i=0}^n [y_i - (b+c\cos(x_i))]\cos (x_i) $$ The way you wrote up the derivatives, turning them into equations half-way through writing, is a mess.

If you are going to split the sum, do it correctly: $\sum_{i=0}^n b = (n+1)b$, not $nb$. Alternatively, you could begin indexing with $i=1$, not with $i=0$.

Keeping the present notation, I get the system $$ (n+1)b +c \sum_{i=0}^n \cos(x_i) = \sum_{i=0}^n y_i $$ $$ b\sum_{i=0}^n \cos(x_i) +c \sum_{i=0}^n \cos^2(x_i) = \sum_{i=0}^n y_i \cos (x_i) $$ Here $n=3$, so $n+1=4$. You have $\sum_{i=0}^n (x_i)$ in place of $\sum_{i=0}^n \cos(x_i)$.

Answer 2

The solution of @user147263 relies on calculus. Another option is to exploit linear algebra.

Form linear system

Start with a series of data points $(x_{k}, y_{k})_{k=1}^{m}$, and the trial function $$ y(x) = c_{1} + c_{2} \cos x, $$ We have the linear system $$ \begin{align} A c & = y \\ \left[ \begin{array}{cc} 1 & \cos x_{1} \\ 1 & \cos x_{2} \\ \vdots & \vdots\\ 1 & \cos x_{m} \end{array} \right] \left[ \begin{array}{c} c_{1} \\ c_{2} \end{array} \right] & = \left[ \begin{array}{c} y_{1} \\ y_{2} \\ \vdots \\ y_{m} \end{array} \right] \end{align} $$ Find the solution vector $c$ which minimizes the sum of the squares of the residuals: $$ r^{2}(c) = \lVert A c - y \rVert_{2}^{2} = \sum_{k=1}^{m} (y_{k} - c_{1} - c_{2} \cos x_{k})^2. $$

Normal equations: Form the normal equations $$ A^{T}A c = A^{T}y. $$

Solve linear system

The solution vector is $$ c = (A^{T}A)^{-1} A^{T}y. $$ Defining the $m-$vector ${\mathbf{1}} = \left[ \begin{array}{c} 1 \\ 1 \\ 1 \\ 1 \end{array} \right]$, and the the $m-$vector $\eta = \left[ \begin{array}{c} \cos x_{1} \\ \cos x_{2} \\ \cos x_{3} \\ \cos x_{4} \end{array} \right]$, the system matrix has the column vector form $A = \left[ \begin{array}{c} \mathbf{1} & \eta \end{array} \right]$, and the product matrix is $$ A^{T} A = \left[ \begin{array}{cc} \mathbf{1} \cdot \mathbf{1} & \mathbf{1} \cdot \eta \\ \eta \cdot \mathbf{1} & \eta \cdot \eta \end{array} \right] = % \left[ \begin{array}{cl} m & \sum_{k=1}^{m} \cos x_{k} \\ \sum_{k=1}^{m} \cos x_{k} & \sum_{k=1}^{m} \cos^{2} x_{k} \end{array} \right] $$ with inverse $$ ( A^{T} A )^{-1} = (\det (A^{T}A))^{-1} \left[ \begin{array}{rc} \eta \cdot \eta & -\mathbf{1} \cdot \eta \\ - \mathbf{1} \cdot \eta & m \end{array} \right] $$ where the determinant is $$ \det (A^{T}A) = m (\eta \cdot \eta) - (\mathbf{1}\cdot \eta)^{2}. $$

For the data set above, $x = \left[ \begin{array}{l} 0 \\ 0.5 \\ 1 \\ 1.5 \end{array} \right]$, $\ $ $\eta = \left[ \begin{array}{l} 1 \\ 0.877583 \\ 0.540302 \\ 0.0707372 \end{array} \right]$, $\ $ $y = \left[ \begin{array}{c} 6.1 \\ 5.4 \\ 3.9 \\ 1.6 \end{array} \right]$, $\ $ and $\mathbf{1} = \left[ \begin{array}{c} 1 \\ 1 \\ 1 \\ 1 \end{array} \right].$

The inner products are $$ \begin{align} \mathbf{1} \cdot \mathbf{1} &= 4, \\ \mathbf{1} \cdot \eta = \eta \cdot \mathbf{1} &= 2.48862, \\ \eta \cdot \eta & = 2.06708. \end{align} $$ The matrices are $$ A = \left[ \begin{array}{rl} 1 & 1 \\ 1 & 0.877583 \\ 1 & 0.540302 \\ 1 & 0.0707372 \end{array} \right], \quad % A^{T}A = \left[ \begin{array}{rl} 4 & 3 \\ 3 & 3.5 \end{array} \right], \quad % (A^{T}A)^{-1} = \det (A^{T}A)^{-1} \left[ \begin{array}{rr} 2.06708 & -2.48862 \\ -2.48862 & 4 \end{array} \right]. $$ The determinant is $\det (A^{T}A) = 2.07509$. The solution vector is $$ c = \left[ \begin{array}{c} c_{1} \\ c_{2} \end{array} \right] = \left[ \begin{array}{r} 1.27258 \\ 4.78565 \end{array} \right] $$

Check result

The solution curve is plotted against the data points.

A contour plot showing $r^{2}$ as a function of the solution vector $c$ with a white dot marking the solution.