Linear, quadratic and exponential regression

Question

I know the formulas for linear and quadratic regression. Please tell me 1) how to model an equation for exponential regression? 2) if I can use the gradient-point formula for linear regression and similar methods for finding equation for quadratic and exponential regression?(solving for a,b,c in y=ax^2 + bx + c and y=ae^(bx)) 3) regression in detail?]

Answer 1

When you perform regression, you face two kinds of models

• Linear models are those where $\frac{dy}{dp_k}$ are independent of all $p_i$'s. The model could be off any apparent complexity such as $$y=a+bx^\pi+c\log(x)+d e^{-x\sqrt 2}$$ Defining variables $t_1=x^\pi$, $t_2=\log(x)$, $t_3= e^{-x\sqrt 2}$ makes the model to be $y=at_1+bt_2+ct_3$ and then the problem is simple using matrix formulation.

• Nonlinear models are those where $\frac{dy}{dp_i}$ dependent of some of the $p_k$'s. For example $$y=a+bx^f+c\log(x)+d e^{-x\sqrt 2}$$ is one case since parameter $f$ has to be tuned. This implies nonlinear regression and in general, "reasonbale" estimates are required.

The case to the exponential function $y=a e^{bx}$ is interesting; it is nonlinear since $$\frac{dy}{da}=e^{bx}\quad \quad \quad\frac{dy}{db}=abe^{bx}$$ but the estimates of the parameters can easily be obtained since $\log(y)=\log(a)+bx$ So, defining $z_i=\log(y_i)$ makes the transformed model to be$z=\alpha+bx$. A first linear regression will provide $\alpha$ and $b$; so, you can start the nonlinear regression work using as estimates $e^\alpha$ and $b$. But, even if the model has been made linear by some transform as here, you must continue because the first step minimizes the sum of the squares on the $z$'s while you want to minimize the sum of the squares on the $y$'s.

I hope that helps. If you want to go deeper, just post.

Edit

Say that you have $n$ data points$(x_i,y_i)$ and you want to fit the model $y=a\,e^{bx}$, that is to say that you want to minimize the sum of the squares of the residuals $$SSQ=\sum_{i=1}^n \left(a\,e^{bx_i}-y_i \right)^2$$ Computing the partial derivatives $$\frac{d\,SSQ}{da}=2\sum _{i=1}^n e^{b x_i} \left(a e^{b x_i}-y_i\right)$$ $$\frac{d\,SSQ}{db}=2 \sum _{i=1}^n a x_i e^{b x_i} \left(a e^{b x_i}-y_i\right)$$ Now, since we want a minimum, these derivatives must be equal to zero.

As you can see, these are nonlinear equations. However, we can eliminate $a$ from the first equation and get $$a=\frac{\sum_{i=1}^n y_i e^{b x_i}}{\sum_{i=1}^n e^{2b x_i}}$$ and we are let with one equation for one unknow $b$. After some simple manipulations, this equation can write $$\Phi(b)=\left(\sum_{i=1}^n y_i e^{b x_i}\right)\left(\sum_{i=1}^n x_i e^{2b x_i}\right)-\left(\sum_{i=1}^n x_iy_i e^{b x_i}\right)\left(\sum_{i=1}^n e^{2b x_i}\right)=0$$ but it does not show analytical solution and numerical methods such as Newton will solve the problem since, remember, the first step gave us an estimate which, except if data contain a large noise, would be quite good.

Another solution is to plot the function $\Phi(b)$ as a function of $b$ and to look for its zero.

All of that assume that you have no regression software.

I hope and wish that this clarify a little.

Edit

If you look for something simpler but not rigorous, have a look at this page. It is much better than the linearized model (as the given plot shows) but slightly worse than the rigorous one I described.

Edit

For illustration purposes, I used the ten following points $$\left( \begin{array}{cc} x & y \\ 1 & 96 \\ 2 & 75 \\ 3 & 60 \\ 4 & 48 \\ 5 & 36 \\ 6 & 30 \\ 7 & 21 \\ 8 & 18 \\ 9 & 12 \\ 10 & 9 \end{array} \right)$$ The first step leads to $\log(y)=4.87206 -0.259481 x$, giving as estimates $a=130.59$ and $b=-0.26$. Starting with these values, $\Phi(b)$ cancels for $b=-0.24435$ (Newton method converges to this value using two iterations) to which corresponds $a=123.235$; to these results corresponds $R^2=0.999413$ which reveals a very good fit. Below are reproduced the data and the recomputed $y$'s $$\left( \begin{array}{ccc} 1 & 96 & 96.5195 \\ 2 & 75 & 75.5954 \\ 3 & 60 & 59.2073 \\ 4 & 48 & 46.372 \\ 5 & 36 & 36.3192 \\ 6 & 30 & 28.4457 \\ 7 & 21 & 22.279 \\ 8 & 18 & 17.4492 \\ 9 & 12 & 13.6665 \\ 10 & 9 & 10.7038 \end{array} \right)$$