Non-linear regression, new try at exponential family problem.

Question

So I am trying to find some kind of an "easiest to understand problem" which is not very easy to solve with anything less than a quite sophisticated optimization framework. I will still of course welcome other solutions as well.

The idea is to find some way to do regression on:

$$f(x) = c_0+\exp[c_3x^2+c_2x+c_1], \\c_0,c_1,c_2,c_3\in \mathbb C, \\x\in \mathbb R$$

estimating $c_0,c_1,c_2,c_3$ parameters from linearly spaced data $x_k = kx$ and we know $(x_k,f(x_k)+e_k)$ potentially with some noise e.

---

Some famous (and useful) special cases of $f$ we want to be able to find:

$c_3=0,c_2\in \mathbb R$ gives exponential model offset by $c_0$, for example how money changes over time at constant interest if $\log c_1$ is in bank and $c_0$ is saved in piggy bank.

$c_3=0,c_2=\lambda i+b$ gives sinusoid of constant frequency and $b$ decides if it spirals.

$c_3=\lambda i,c_2=0$ gives a complex quadrature chirp of constant acceleration $\lambda$, like in a previous question which had (too) easy solution.

Answer 1

$$f(x) = c_0+\exp\left(c_3x^2+c_2x+c_1\right) $$

A straightforward method (no initial guess, no iteration) is shown below.

In order to comply with the symbols used in the sheet below, one have to change the notations : $$\begin{cases} f(x)=y(x)\\ a=c_0 \\ b=e^{c_1} \quad\to\quad c_1=\ln(b)\\ p=c_3 \\ q=c_2 \end{cases}$$ $$y(x)=a+b\,e^{p\,x^2+q\,x}$$ This is a copy from a French paper.

COMMENT :

The method is based on the fitting of an integral equation. In the present case, the integral equation is : $$y=2p\int xy\,dx+q\int y\,dx-apx^2-aqx+c$$ which comes from the integration of the function $y(x)=a+b\,e^{p\,x^2+q\,x}$.

For more explanation on the theory, see the paper : https://fr.scribd.com/doc/14674814/Regressions-et-equations-integrales

The numerical integrations (calculus of $S_k$ and $T_k$) introduces some additional deviations if the number of points is too small.

The result is not exactly the least mean square deviation with respect to $y(x)$ but is a least mean square deviation with respect to the integral equation.

If the result is not sufficiently accurate, on cannot avoid to use a non-linear method of regression which requires some initial guess of the parameters. The above method gives approximates of the parameters which can be used as very good "guess".

Answer 2

Warning: This is a hacky solution :D.

We have the model equation

$$y = \alpha+\exp[ax^2+bx+c], \\\alpha,a,b,c\in \mathbb C, \\x\in \mathbb R.$$

In the first step, we create a grid (structured or better random) of possible values for $\alpha$ and from this point assume that $\alpha^{[k]}$ is fixed, in which $k$ is the index of the $k^{\text{th}}$ possible value for $\alpha$ that we selected. Then we introduce a new dependent variable of $\exp(z^{[k]})=y-\alpha^{[k]}$. This will allow us to obtain the simplified regression equation

$$z^{[k]}=a^{[k]}x^2+b^{[k]}x+c^{[k]}.$$

This problem can be solved by a simple least squares regression. Then out of all the $k$ possible values for $\alpha^{[k]}$ we choose the one with the smallest test error. After having obtained this solution you can further refine your grid for $\alpha$ to obtain a better solution. In this method, I treated $\alpha$ as a hyperparameter instead of using it as a coefficient.

Note, that by this method we do not have any guarantees that the solution does globally minimize the sum of squared errors (if we use the squared error as our loss function). As the values for $y$ can be complex, we would need to use the sum of the squared magnitude of the error as our loss function.