Consider $$f(x,y,k) = kx-y$$ and its level set $f(x,y,k) = 0$
Now for example assume that I have found a point on this level set $(9,81,9)$, how can I then estimate $k$ in $(8,76,k)$? This translates to if we know $81/9=9$, how can we calculate $76/8 = ?$
I can calculate gradient $$\nabla f = [k,-1,x]$$
To remain on surface (locally), I shall go perpendicular to gradient. So $$[\Delta_x,\Delta_y,\Delta_k][k,-1,x]^T = 0$$
Now this lets us derive a wealth of algorithms which we can iterateively take small steps:
- Take step along a line in the tangent plane.
- Make correction in gradient direction to come back to surface.
But the problem I hit is when I want to go one step further into higher order models.
For example if I want to use a second order polynomial model...
All second order partial differentials are zero!
How can I modify $f$ or my algorithm to somehow add information of higher order than $1$ ?
Own work:
I've been thinking I can do $f \to f^3$. This is sure to generate a higher order power series expansion everywhere, right?
As @TedShifrin writes, of course we do get second order partial derivatives:
$$\frac{\partial^2}{\partial k \partial x} f = \frac{\partial^2}{\partial x \partial k} f = 1$$
So we get a non-zero, but constant Hessian:
$${\bf H}\{f\} = \begin{bmatrix}0&0&1\\0&0&0\\1&0&0\end{bmatrix}$$
Perhaps even one of the simplest and sparsest Hessians possible.
We can now go on to devise (for example) a second order polynomial approximation as per this question:
$$f(\mathbf{x}^* + h\mathbf{y}) = f(\mathbf{x}^*) + h \nabla f(\mathbf{x}^*)^T \mathbf{y} + \dfrac{1}{2} h^2 \mathbf{y}^T \nabla^2 f(\mathbf{x}^*) \mathbf{y} + O(h^3)$$
If we explicitly write $\bf g$ and $\bf H$ for gradient and Hessian:
$$f(\mathbf{x}^* + h\mathbf{y}) = f(\mathbf{x}^*) + h {\bf g}^T \mathbf{y} + \dfrac{1}{2} h^2 \mathbf{y}^T {\bf H} \mathbf{y} + O(h^3)$$
Now we can use this however we see fit to pick out a better direction for estimation.