Does anyone know of a text that treats the problem of gradient descent from a continuous perspective instead of a discretized perspective?
For example, most text investigates the numerical properties of
$$x_{k+1} = x_k - \alpha_k \nabla f(x_k)$$ (i.e. Luenberger et. al.)
I am interested whether there is a text that investigates the property of
$$\dot x = -\nabla f(x)$$
from a purely continuous perspective
Thanks!
I guess the easiest option would be to apply Lyapunov theory:
https://en.wikipedia.org/wiki/Lyapunov_function
https://en.wikipedia.org/wiki/Lyapunov_stability
As a short summary: take $f(x)$ as a candidate Ljapunov function, which puts some reasonable restrictions on the shape of your optimization landscape (i.e. locally positive-definite function). All you then have to do is to proof that $\frac{d}{dt}f(x)=\nabla f(x) \frac{d}{dt}x(x)=-(\nabla f(x))^2< 0$ for $x\neq 0$, where $x=0$ is your optimum.
Then you know that $x=0$ is a (locally) assymptotic stable steady state, and thus, that $x\rightarrow 0$.
The method is called "Lyapunov's second method for stability". It's pretty old and well established, thus, just say that you use Lyapunov's second method and it should be fine. His original paper is in Russian and I couldn't find it in the Inet ~5 years ago, but there is a French translation available if you need a reference (around 1900, I think). Otherwise, follow the references in the wikipedia article.