Suppose that I have two bodies in 2D space, each with a local coordinate system. The problem is to find the configuration variables (orientation and position for each body) such that the angle between the local y-axes of the two bodies is zero. The objective function is then just the angle between the two y-axes. Please ignore the fact that this is solvable directly without iteration.
Now suppose that at my starting guess for the configuration variables the angle between the y-axes is ±π. In other words, I am starting at the maxima. At this point the partial derivatives will all be zero and so the solver cannot determine the step direction.
However, using knowledge of the problem, we know that despite the zero partial derivatives, we are not at the minimum. What possible solutions are there to proceed? Can we just set the partial derivatives to some reasonably small non-zero value, causing the solver to at least step away from the stationary point. What effect will this have on quasi-Newton methods like BFGS or L-BFGS, which use the partial derivative history to determine the step direction?