Least Square Approximation for Exponential Functions

Question

I'm a little confused on how to approach the following problem.

Use the least square method to fit a curve of the form $y=a\cdot b^x$ to a collection of $n$ data points $(x_1,y_1),...,(x_n,y_n)$ Find $a$ and $b$.

From what I understand. Given data {$(x_1,y_1),...,(x_n,y_n)$}, we may define the error associated to $y=a\cdot b^x$ by \begin{equation} E(c,d)=\sum_{i=1}^{n}(y_i-a\cdot b^{x_i})^2 \end{equation} The goal is to find values of $a$ and $b$ that minimize the error. In multivariable calculus we learn that this requires us to find the values of $(a,b)$ such that

\begin{array} d \frac{\partial E}{\partial a}=0 & & \frac{\partial E}{\partial b}=0 \end{array}

Differentiating $E(a,b)$ yields:

\begin{eqnarray} \frac{\partial E}{\partial a}&=&\sum_{i=1}^n2(y_i-a\cdot b^{x_i})(-b^{x_i})=0\\ \frac{\partial E}{\partial b}&=&\sum_{i=1}^n2(y_i-a\cdot b^{x_i})(-a\cdot b^{x_i-1}x_i)=0\\ \end{eqnarray}

I know that the least square method tries to minimize the error, but I can't help but feel that I doing this somewhat wrong since this isn't a polynomial - but an exponential - function.

The alternative was for me to take the natural log of $y=a\cdot b^x$ to get: \begin{equation} \ln{y}=\ln{a}+x\ln{b} \end{equation} Then defining $\ln{y}=Y$,$\ln{a}=A$ and $\ln{b}=B$ to get: \begin{equation} Y=A+xB \end{equation} Then from here apply the least square method to my new linear equation.

Any suggestions or feedback on what I can do to solve this problem?

---

Thank you for your time.

Answer 1

The standard approach to solving this question is indeed to linearize by taking the logarithm. The same trick is used for the power law, $y=a\cdot x^b$, among others. This is probably the approach expected by your professor.

Anyway this is a twist as the nonlinear transform does not provide the true minimum solution of the original problem, as the errors themselves are rescaled non-uniformly.

Your formulation is indeed the correct one, but you end-up with a nasty system of equations that has no analytical solution.

So you must resort to a numerical approach such as that of Levenberg and Marquardt which solves these nonlinear equations iteratively. This method requires a good initial approximation, for which one will naturally chose the solution obtained from linearization.

Answer 2

You are perfectly correct making the problem linear. But this only gives you estimates of parameters $a$ and $b$.

Using those, you have to perform a nonlinear regression since what is measured is $y_i$ and not $\log(y_i)$. The first steps minimizes $$\sum_{i=1}^n \big(A+Bx_i-\log(y_i)\big)^2$$ while you need, as you wrote, to minimize $$E=\sum_{i=1}^n \big(a b^{x_i}-y_i\big)^2$$ So, $A,B$ (given by the linear regression) provide estimates of $a,b$ from which you can start the minimization.

Since the estimates would probably be good, you also could use Newton-Raphson method to solve $$\frac{\partial E}{\partial a}=\frac{\partial E}{\partial b}=0$$ with numerical derivatives. For sure, if you have an optimizer, the problem is simple (as Yves Daoust mentioned).