Does the Least Squares Regression Method work for any line type?

Question

I recently learned how to apply the least squares method to do linear regression. I also understand that it can be used for quadratic regression, by minimizing the error for three variables, two coefficients and a constant, instead of two variables. Would the same method apply to most, or all, types of equations? Could I simply assume coefficients wherever possible, and a constant, then find the partial derivative with respect to each, then set them equal to zero and solve? For example, could I regress to *a*log(*b*x)+c? Could I use logarithms, sine waves, exponential function, etc? If not, what are the exceptions? Where is this method not possible? Why?

Thanks in advance for all responses.

Answer 1

My above comment: Yes, but then you have to distinguish between linear and non-linear least squares. Both of these are solved differently, depending on the nature of the relationship.

In response to your question of an example, some are given below. Note that I will use $a,\,b$ and $c$ as the coefficients to be determined, $x$ as the independent/predictor variable and $y$ as the dependent/response variable.

Linear examples: $$\begin{align} y&=a+bx\\ \ln y&=a+b\ln x \quad(\text{equivalent to the nonlinear form } y=e^ax^b)\\ y^2&=a+bx^2-ce^x \end{align}$$ These are linear because the equations are linear in the unknown coefficients.

Nonlinear examples: $$\begin{align} y&=ax^b+c\\ y&=a\sin\left(bx+c\right)\\ y&=\frac{x+a}{x+b}\quad(\text{equivalent to the linear form } ay-b=x-xy) \end{align}$$ These are non-linear because the equations are non-linear in the unknown coefficients.

Answer 2

Yes you can use any function, but the results are not guaranteed to make sense. Also, you have to have at least as many points as parameters.

If the function is linear in the parameters, then you get a linear system of equations, which can be solved by usual methods.

If the function is non-linear in the parameters, then you get a non-linear system of equations that has to (usually) be solved by iteration. In this case, it is quite important to get a good starting point. It is also probably not worth your while to write your own solver, as there are a number of non-linear least squares solvers available.

Answer 3

A very important comment Daryl has made. It took me years to realize this and not be confused. Linear regression means the fitting form must be linear in the unknowns (for example, $a,b,c,$ that you are trying to solve for). You can have non-linearity in the independent variable $x$. So for

$$y=ax^2+bx+c$$

you would still use linear least squares fit and yes it is still called linear. This isn't quadratic squares fit or something because $a,b,c$ are still linear. If you wanted to work with something like

$$y=\frac{ax+b}{cx+d}$$

or even something like

$$y=\frac{a}{a+b}x$$

now you cannot use linear least squares fit. Now you have to use non-linear least squares fit. The second one is even linear in $x$ but nonlinear in $a,b$ so have to use non-linear least squares. So generally there are only two categories, linear and nonlinear least squares fit.

So the answer to your question is, the non-linearity in $x$ doesn't matter as long as you have linearity in the unknown coefficients so yes sine,log, exponential, any degree polynomials in $x$ are all good. You can use linear least squares on them.

But on the other hand, the example you gave $a\log(bx)+c$ is non-linear in $b$ so you have to use NLLSF on this form.

By the way, NLLSF is a weeeee bit more complicated than LLSF. Very interesting subject in its own right but will require a bit more math background. You can have things like no solutions or a huge number of finite or even infinite solution. Usually the methods aren't exact. They will be iterative meaning you cannot get an exact answer in one shot. You only get an approximation. Sometimes you need to provide a starting guess. Then you run the iteration again to get a better approximation and then again and stop whenever you think your answer is good enough. You can converge to a different solution too depending on where you started from.

LLSF is rather easy, no initial guess required, the answer is guaranteed (given enough points), the answer will be unique, the answer will be exact in one iteration, and there are fast ways of doing it too. The only "bad" thing I find annoying is that you already need to know the form of the fitting function like is it a polynomial or a trig function or a logarithm in advance. But you need that with NLLSF too.