As an example, let's consider non-linear least squares (NLLS), which can be solved using Levenberg-Marquardt (LM). LM can be used to solve many different NLLS objective functions, but the LM constants need to be tuned to achieve fast convergence to a local minimum.
Would it be possible to handcraft an optimization algorithm tailored to a specific NLLS objective function $E$, which would not get trapped in saddle points and local optima as much as LM does for the specific function $E$?
My line of thinking is that different NLLS functions can have different types of nonlinearities and as we know from no-free-lunch theorem, there isn't a single optimization algorithm that works best for all possible optimization problems.
Such tailored algorithm perhaps couldn't be used for any other specific objective function $F$ (or it would perform poorly when optimizing $F$), but it would work extremely well for optimization of $E$.
This handcrafted algorithm could be devised perhaps by analyzing the structure of the non-linear function $E$ and coming up with an optimization scheme, which would exploit the geometry of nonlinearities specific to $E$ (maybe by analyzing the plot of the function?).
Do such algorithms, which are tailored to a specific function, exist? And if they do, how would I go about deriving a specialized optimization algorithm for a specific (NLLS or not) function?