What is Directional Curvature and how can I achieve it for any function? A common approach with an example would be much appreciated.
(Reference: I am reading "The Non-convex Geometry of Low-rank Matrix Optimization" paper and in section 1.2, Weighted PCA part, I got stuck. Link to the paper: https://academic.oup.com/imaiai/article/8/1/51/4951409)
Thanks in advance!
$\newcommand{\R}{\mathbb{R}}$The authors give the definition of directional derivative, but I guess you need to know why this is defined like that. Consider a function $f:\R^n\to\R$. At a point $x_0$, the gradient of $f$, $\nabla f(x_0) \in \R$ is vector valued and its Hessian is matrix valued $\nabla^2 f(x_0) \in \R^{n\times n}$.
In other words $\nabla^2 f$ maps a point $x_0$ to a matrix, that is, a linear operator, which, in turn, maps from $\R^n$ to $\R^n$, that is
$$ \nabla^2 f: \R^n \to L(\R^n, \R^n). $$
Recall that the directional derivative of $f$ along the direction $d$ is
$$ \nabla_d f(x_0) := \langle \nabla f(x_0), d\rangle = \nabla f(x_0)^\top d. $$
The second-order directional derivative is
$$ \nabla_{d,d}^2 f(x_0) := \langle \nabla^2 f(x_0)(d), d\rangle, $$
where $\nabla^2 f(x_0)(d)$ is the Hessian matrix of $f$ at $x_0$ evaluated at $d$ (recall that the Hessian of $f$ at $x_0$ is a linear operator, so it maps a vector $d$ to a vector \nabla^2 f(x_0)(d)). Equivalently, $\nabla^2 f(x_0)(d) = \nabla^2 f(x_0)\cdot d$, so $\nabla_{d,d}^2 f(x_0) = d^{\top} \nabla^2 f(x_0) d$.
The authors of the paper you referenced denote this by $[\nabla^2 f(X)](D, D)$.