Constrained parameters in non linear curve fitting

Question

I have to fit some data points to the non linear curve defined by:

$$y(x) = a + \frac{b}{(x - c)}$$

with constraints $a,b,c \geq0$

So far I've tried gradient descent with penalty methods but I'm not getting correct results. At any point $(x_0,y_0)$, my loss function looks like:

$$(y_0 - y(x_0))^2 -r (\log(a) + \log(b) + \log(c))$$

What's the best way to model this?

I was going through Constrained parameters in least square curve fitting, which is similar to my problem. How can I use this approach with the curve I defined?

Answer 1

What I'm going to propose is definitely not common but it will definitely help with data that is not very close to the singularity $c$, since it penalizes them more than the conventional approach to the loss function and therefore makes them more visible to the minimization procedure. As an example, if you want to fit a number of points on the tail of the curve, then finding the value $c$ is certainly unstable, since all the data points you have have a similar loss function and thus the parameters a,b,c will have similar loss functions for a wide range of values, making it hard to distinguish what the right choice is, especially if the problem is non-linear and has several minima.

Consider the loss function

$$L=\sum_i(y_i(x_i-c)-a-b(x_i-c))^2$$

If it is zero, then we recover that the (x,y)'s are on the same curve you wish to use. It is a fortunate coincidence that the minimization problem of this loss function has a unique solution!

$$\frac{\partial L}{\partial a}=\sum_i (y_i(x_i-c)-a-b(x_i-c))=0\\ \frac{\partial L}{\partial b}=\sum_i (y_i(x_i-c)-a-b(x_i-c))(-x_i+c)=0\\ \frac{\partial L}{\partial c}=\sum_i (y_i(x_i-c)-a-b(x_i-c))(-y_i+b)=0$$

We can rewrite these equations in the form (note that $\overline{WXY}=\frac{\sum_i W_iX_iY_i}{N}$):

$$a+b\bar{x}+c\bar{y}-bc-\overline{xy}=0\\ a\bar{x}+b\overline{x^2}+c\overline{xy}-bc\bar{x}-\overline{yx^2}=0\\ a\bar{y}+b\overline{xy}+c\bar{y^2}-bc\bar{y}-\overline{y^2x}=0$$

After a miraculous cancellation of the non-linear term $bc$ we obtain the unique solution

$$b=\frac{AZ-BY}{XZ-Y^2}\\ c=\frac{BX-AY}{XZ-Y^2}\\ a=-b\bar{x}-c\bar{y}+bc+\overline{xy}$$

where I defined:

$$A=\overline{x^2y}-\overline{xy}~\overline{x}~~,~~B=\overline{y^2x}-\overline{xy}~\overline{y} \\ X=\overline{x^2}-\bar{x}^2~~, ~~Z=\overline{y^2}-\bar{y}^2~~, ~~ Y=\overline{xy}-\bar{x}\bar{y}$$

With this now you have a tool to minimize with analytical control. Hope it works!

Answer 2

Preliminary inspection with the data :

x = [0.000000000050, 0.000000000075, 0.000000000125, 0.000000000175, 0.000000000275, 0.000000000475, 0.000000001100]

y = [0.791999995708, 0.703999996185, 0.659999996424, 0.615999996662, 0.571999996901, 0.527999997139, 0.483999997377]

First I suspected a computation problem due to the low magnitude of the x values.

In order to eliminate any trouble of this kind, the change of variable $\quad X=10^{10}x\quad $ transforms the data into :

X = [0.50, 0.75, 1.25, 1.75, 2.75, 4.75, 11.00]

$y=a+\frac{b}{x-c}\quad$ is transformed into $\quad y=a+\frac{B}{X-C}\quad \begin{cases} B=10^{10}b\\ C=10^{10}c \end{cases}$

A least mean square regression gives : $\begin{cases} a= 0.45651934\\ B= 0.37824659\quad ;\quad b=0.000000000037824659\\ C=-0.66325\quad ;\quad c=-0.000000000066325 \end{cases}$

So, the parameter $C$ is negative and this is not due to numerical computation with small numbers.

Thus the first hypothesis is eliminated.

By comparison, if we proceed with linear regression with respect to the parameters $a$ and $b$ for given values of $C$ we observe how the fitting depends on $C$ :

The deviation is continuously increasing when $C$ is increased in order to tend to positive value.

As a consequence if the regression is subjected to a constraint $\quad c\geq 0\quad$ one have to accept a very bad fitting.

It is normal that your results from gradient descent with penalty methods are not as good as expected. This is the unavoidable consequence of the constraint in the specific case of the function $y=a+\frac{b}{x-c}$.

One can expect a better fitting with constraint in choosing another function, may be with more parameters and/or better convenient as model.