Consider a Riemannian manifold $(M,g)$, which we assume geodesically complete. Given a smooth function $f : M \to \mathbb{R}^+$. Assume that $f$ has a global minimum on $M$. I am interested in the following optimization problem : $$ \text{Find } p^{\ast} \in \mathop{\mathrm{argmin}} \limits_{p \in M} f(p). $$ Let $\nabla^{g}f$ denote the gradient of $f$ with respect to the Riemannian metric $g$. Let $p \in M$. By definition, the gradient of $f$ at $p$, denoted by $\nabla^{g}f(p)$ is the vector in $\mathrm{T}_{p}M$ which satisfies to : $$ \forall v \in \mathrm{T}_{p}M, \, df_{p}(v) = \left\langle \nabla^{g}f(p),v \right\rangle_{p}$$ The critical points of $f$ are the points $q \in M$ such that $\nabla^{g}f(q)=0$.
If $M$ was the Euclidean space $\mathbb{R}^n$, one could start by finding the critical points of $f$ and then check whether a given critical point is a minima or a maxima of $f$. My question is : why can we also proceed like this for Riemannian manifolds ? In other words : what ensures that the extrema of $f$ are among the critical points of $f$ ? Is it based on a Talyor expansion ? This point is not clear to me.
You are absolutely right, we do the same on manifolds as in Euclidean space. The local extrema are among the critical points of $f$ on the manifold $M$. This comes from the fact that if a point is a critical point in one coordinate chart around the point, then it is a critical point in all cooridnate charts around that point. Furthermore, the Hessian of a critical point (the matrix of second derivatives at the point) changes as a bilinear form from chart to chart, so its signature is independent on coordinates and is thus well defined on the manifold. Because of that Taylor expansion applies and if it works in one chart, it works in all. If you have a positive definite Hessian, then the point is a local minimum. If it is negative definite -- a local maximum. In fact, this is what Morse theory is based upon. And this is independent on any metric on the manifold. The metric only helps you define the gradient vector field.