Suppose I am trying to fit a non-linear function with parameters A, and B to some observed data:
$$f(x,A,B) = Ae^{(Bx)} $$
To the best of my understanding Jacobian matrix is a matrix containing the first derivative of each variable, in this case parameters A and B.
$$J = \begin{bmatrix} \partial{f}/\partial{A} & \partial{f}/\partial{B} \end{bmatrix} $$
However, I am not including the independent variable, x, in the Jacobian.
When performing gradient descent via Newton's Method for Optimisation, the formula for the updated values of A, and B is :
$$ \mathbf{x}_{n+1} = \mathbf{x}_{n} -[\mathbf{H}f(x)]^{-1}\nabla f(x) $$
Where $\mathbf{x}_{n}$,$\mathbf{x}_{n+1}$ contain the current and updated values for A and B for iteration $n$.
I am defining $\nabla f(x) $ as:
$$\nabla f(x) = -2J^{T}(Y-f(x,A,B))$$
Where Y is the observed data (X x 1 matrix), and $f(x,A,B)$ is the model with the current values of the parameters. $J$ is a (2 x obvs) matrix that includes the derivatives of $f$ with respect to A and B calculated at every point, $x$.
My question is whether it is correct to calculate the derivative with respect to A, B at every point x for the Jacobian, J. If this is correct, then how do I calculate the Hessian matrix, $[\mathbf{H}f(x)]$? Do I instead include a derivative with respect to x, even though x will not change with each iteration, as it is an independent variable?
Many thanks for your assistance.
You seem to be confused about the difference between a loss/objective function (usually denoted $L$) and the parameteric function $f$ you are fitting to.
A loss function measures the error of your fit. For example, you could take the loss function $L$ to be mean squared error: $$ L(A,B)\equiv\frac{1}{2}\sum_{i}\left(f(x_{i};A,B)-y_{i}\right)^{2}\equiv\frac{1}{2}\sum_{i}\left(Ae^{Bx_{i}}-y_{i}\right)^{2}. $$ One way to find a local minimizer $(A,B)$ of $L$ is to solve $\nabla L(A,B)=0$. This is where Newton's method, a tool for finding roots of nonlinear systems, comes in handy. In general, Newton's method applied to solve the equation $F(x)=0$ produces iterates $$ x_{n+1}=x_{n}-[JF](x_{n})^{-1}F(x_{n}) $$ where $JF$ is the Jacobian of $F$. Taking $F\equiv\nabla L$ in the above, we get $$ (A_{n+1},B_{n+1})=(A_{n},B_{n})-\left([J\nabla L](A_{n},B_{n})\right)^{-1}\nabla L(A_{n},B_{n}). $$ You should try to prove that $J\nabla L=HL$ where $HF$ is the Hessian of $F$.