Minimise this function?

Question

I'd like to minimise a function like:

$$ R^2(x) = \sum_{i=1}^N (Y_i - \frac {a_i}{x \cdot c_i+b_i})^2 $$

This is similar to Ordinary Least Squares, but seems to be harder to solve because the $x$ is in the denominator.

Can this be solved?

Answer 1

You have $n$ data points $(a_i,b_i,c_i,y_i)$ and you want to fit the nonlinear model $$y=\frac a {x\, c + b}$$ where $x$ is a parameter to be tuned.

So, as you wrote, the least square procedure will consist in the minimization of $$F(x)=\frac 12\sum_{i=1}^n r_i^2 \qquad \text{where} \qquad r_i=y_i-\frac{ a_i} {x\, c_i + b_i}$$ which will be solved looking for the zero of function $$G(x)=F'(x)=\sum_{i=1}^n r_i \frac {dr_i}{dx}\qquad \text{where} \qquad \frac {dr_i}{dx}=\frac{ a_i\,c_i} {(x\, c_i + b_i)^2}$$ This could be achieved using Newton method using $$G'(x)=\sum_{i=1}^n\left( \left(\frac {dr_i}{dx}\right)^2+r_i\frac {d^2r_i}{dx^2}\right)\qquad \text{where} \qquad \frac {d^2r_i}{dx^2}=-\frac{ a_i\,c_i^2} {(x\, c_i + b_i)^3}$$ Starting from a guess $x_0$, Newton method will update it according to $$x_{n+1}=x_n-\frac{G(x_n)}{G'(x_n)}$$ All of that looks simple provided a "reasonable" (not to say good) estimate of the parameter to be tuned.

To get this estimate, assume that the errors are small and rewrite, for the time being, the model as $$\frac 1{y}=\frac {x\, c + b} a\implies \frac a{y}-b=x\,c$$ which is linear with respect to $x$; from this, we can deduce the estimate $$\color{red}{x_0=\frac{\sum_{i=1}^n \left(\frac {a_i}{y_i}-b_i\right)\,c_i } {\sum_{i=1}^n c_i^2 }}$$ At this point, you are ready to go for the fine tuning of the parameter using either nonlinear regression or solving equation $G(x)=0$.

For illustration purposes, let us consider the following data set $$\left( \begin{array}{cccc} i & a_i & b_i & c_i & y_i \\ 1 & 1 & 1 & 1 & 0.22 \\ 2 & 1 & 1 & 2 & 0.13 \\ 3 & 1 & 1 & 3 & 0.09 \\ 4 & 1 & 2 & 1 & 0.18 \\ 5 & 1 & 2 & 2 & 0.11 \\ 6 & 1 & 2 & 3 & 0.08 \\ 7 & 1 & 3 & 1 & 0.15 \\ 8 & 1 & 3 & 2 & 0.10 \\ 9 & 1 & 3 & 3 & 0.07 \\ 10 & 2 & 1 & 1 & 0.45 \\ 11 & 2 & 1 & 2 & 0.25 \\ 12 & 2 & 1 & 3 & 0.18 \\ 13 & 2 & 2 & 1 & 0.37 \\ 14 & 2 & 2 & 2 & 0.22 \\ 15 & 2 & 2 & 3 & 0.16 \\ 16 & 2 & 3 & 1 & 0.31 \\ 17 & 2 & 3 & 2 & 0.20 \\ 18 & 2 & 3 & 3 & 0.15 \\ 19 & 3 & 1 & 1 & 0.67 \\ 20 & 3 & 1 & 2 & 0.38 \\ 21 & 3 & 1 & 3 & 0.26 \\ 22 & 3 & 2 & 1 & 0.55 \\ 23 & 3 & 2 & 2 & 0.34 \\ 24 & 3 & 2 & 3 & 0.24 \\ 25 & 3 & 3 & 1 & 0.46 \\ 26 & 3 & 3 & 2 & 0.30 \\ 27 & 3 & 3 & 3 & 0.22 \end{array} \right)$$ Using the method, we obtain $x_0=3.4941$ and Newton iterates will be $$\left( \begin{array}{cc} n & x_n \\ 0 & 3.494096074 \\ 1 & 3.472142899 \\ 2 & 3.472452615 \\ 3 & 3.472452678 \end{array} \right)$$ which is the solution for ten significant figures.

In fact, the data were generated using $x=3.456$ and rounded to two decimal places.

All of the above can easily be done using Excel.