We consider the least square problem in the case where we got only one independant variable $x_i$ and only one dependant variable $y_i$. The number of observations is $n$.
In the case of the linear fit, we want to estimate $y_i$ with a function $f(x_i,µ) = µ_0 + x_i * µ_1$ under the constraint of minimizing $\sum_i{(y_i-f(x_i,µ))²}$.
The solutions can be expressed in the simple form :
- $µ_1 = \frac{covariance(x_i,y_i)_{i=1..n}}{variance(x_i)_{i=1..n}}$
- $µ_0 = \frac{(\overline{y} - \overline{x} * µ_1)}n$
In the case of the quadratic fit, we got instead $f(x_i,µ) = µ_0 + x_i * µ_1 + x_i²*µ_2$.
Is there a way to express $µ_0$, $µ_1$ and $µ_2$ in an simple form ?
Simplicity is in the eye of the beholder.
Linear Fit
Linear system
$$ % \begin{align} % \mathbf{A} \mu & = y \\ % \left[ \begin{array}{cc} 1 & x_{1} \\ \vdots & \vdots \\ 1 & x_{m} \\ \end{array} \right] % \left[ \begin{array}{c} \mu_{0} \\ \mu_{1} \\ \end{array} \right] &= % \left[ \begin{array}{c} y_{1} \\ \vdots \\ y_{m} \\ \end{array} \right] % \end{align} % $$
Normal equations
$$ % \begin{align} % \mathbf{A}^{*} \mathbf{A} \mu & = \mathbf{A}^{*} y \\ % \left[ \begin{array}{cc} \mathbf{1} \cdot \mathbf{1} & \mathbf{1} \cdot x \\ x \cdot \mathbf{1} & x \cdot x \\ \end{array} \right] % \left[ \begin{array}{c} \mu_{0} \\ \mu_{1} \\ \end{array} \right] &= \left[ \begin{array}{cc} \mathbf{1} \cdot y \\ x \cdot y \\ \end{array} \right] % \end{align} $$
Least squares solution
$$ % \begin{align} % \left[ \begin{array}{c} \mu_{0} \\ \mu_{1} \\ \end{array} \right]_{LS} % &= % \left( \det \left( \mathbf{A}^{*} \mathbf{A} \right) \right)^{-1} % \left[ \begin{array}{rr} x \cdot x & -\mathbf{1} \cdot x \\ -\mathbf{1} \cdot x & \mathbf{1} \cdot \mathbf{1} \\ \end{array} \right] % \left[ \begin{array}{cc} \mathbf{1} \cdot y \\ x \cdot y \\ \end{array} \right] \\ % &= \left( \det \left( \mathbf{A}^{*} \mathbf{A} \right) \right)^{-1} \left[ \begin{array}{cc} % \left( x \cdot x \right) \left( \mathbf{1} \cdot y \right) - \left( \mathbf{1} \cdot x \right) \left( x \cdot y \right) \\[4pt] % \left( \mathbf{1} \cdot \mathbf{1} \right) \left( x \cdot y \right) - \left( \mathbf{1} \cdot x \right) \left( \mathbf{1} \cdot y \right) % \end{array} \right] % \end{align} % $$
Quadratic fit
Linear system
$$ % \begin{align} % \mathbf{A} \mu & = y \\ % \left[ \begin{array}{ccc} 1 & x_{1} & x^{2}_{1} \\ \vdots & \vdots & \vdots \\ 1 & x_{m} & x^{2}_{m} \\ \end{array} \right] % \left[ \begin{array}{c} \mu_{0} \\ \mu_{1} \\ \mu_{2} \\ \end{array} \right] &= % \left[ \begin{array}{c} y_{1} \\ \vdots \\ y_{m} \\ \end{array} \right] % \end{align} % $$
Normal equations
$$ % \begin{align} % \mathbf{A}^{*} \mathbf{A} \mu & = \mathbf{A}^{*} y \\ % \left[ \begin{array}{rrr} \mathbf{1} \cdot \mathbf{1} & \mathbf{1} \cdot x & \mathbf{1} \cdot x^{2} \\ x \cdot \mathbf{1} & x \cdot x & x \cdot x^{2} \\ x^{2} \cdot \mathbf{1} & x^{2} \cdot x & x^{2} \cdot x^{2} \\ \end{array} \right] % \left[ \begin{array}{c} \mu_{0} \\ \mu_{1} \\ \mu_{2} \\ \end{array} \right] &= \left[ \begin{array}{r} \mathbf{1} \cdot y \\ x \cdot y \\ x^{2} \cdot y \\ \end{array} \right] % \end{align} $$
Least squares solution
$$ % \begin{align} % \left[ \begin{array}{c} \mu_{0} \\ \mu_{1} \\ \mu_{2} \\ \end{array} \right]_{LS} % &= \left( \det \left( \mathbf{A}^{*} \mathbf{A} \right) \right)^{-1} \left[ \begin{array}{crc} % \Sigma x^2 \Sigma x^5 - \left( \Sigma x^3\right)^2 & \Sigma x^2 \Sigma x^3 - \Sigma x \Sigma x^4 & \Sigma x \Sigma x^3 - \left( \Sigma x^2\right)^2 \\ % \Sigma x^2 \Sigma x^3 - \Sigma x \Sigma x^4 & m \Sigma x^4 - \left( \Sigma x^2 \right)^2 & \Sigma x \Sigma x^2 - m \Sigma x^3 \\ % \Sigma x \Sigma x^3 - \left(\Sigma x^2\right)^2 & \Sigma x \Sigma x^2 - m \Sigma x^3 & m \Sigma x^2 - \left( \Sigma x \right)^2 \\ % \end{array} \right] % \left[ \begin{array}{cc} \Sigma y \\ \Sigma xy \\ \Sigma x^{2}y \\ \end{array} \right] \\ % \end{align} % $$