Optimizing trigonometric and nonlinear functions

Question

First, Please, keep in mind that I'm a programmer not mathematician, and I have a fair mathematical background.

I used optimization in Java to fit some observations to a trigonometric function, I tried the following optimizers: BOBYQA, CMA-ES, Powell, and Simplex to optimize the function as a scalar function, and also Levenberg-Marquardt and Gauss-Newton to optimize it as a vector function, I got good results for

$z = a.\sin(x)+b.\cos(y)$

, but for a non-linear relation like: $y = \cos(a.x)$ or $z = \sin(a.x) + \cos(b.x)$, the results are very unsatisfying and I have to choose an initial guess very close to the correct values of $a$ and $b$, and certainly this is not practical, example:

$y=6.4\cdot e^{5.5x}$

gives

$(6.39 \text{ and } 5.5)$ if the initial guess is set to $(7,7)$

and gives

$(2.07 \text{ and }5.5)$ for initial guess of $(8,8)$

and , certainly, this is a very bad guess. So I guess I need either of two choices:

1. Find a method for selecting a very good initial guess
2. Find a method for improving the optimizer so that it could get a good result even if the initial guess is not good.

Thanks

Answer 1

I finally got it, the problem was caused by falling in a local optimum, following is a short comparison between results I got from the mentioned algorithms:

• Powell and BOBYQA give wonderful results with high precision but low accuracy, this means that you have to provide a very good initial guess and rather tight bounds
• Simplex is slightly better in not falling in a local optimum, I think this is because it uses Nelder-Mead Simplex or Multi-Directional Simplex to update its guesses, both allows providing long steps to search far from the initial guess and thus avoiding the local optimum, it also requires no bounds, it estimates them well
• CMA-ES is much more accurate but slightly less precise, it allows you to specify bounds and step length, but strangely gives different results for different runs and some of them may fall in a local optimum.

I finally made the following algorithm to get reliable, accurate, and precise results:

1. Use CMA-ES to get a rough estimation with high accuracy
2. Calculate the mean squared error
3. If the mean squared error is larger than a threshold, repeat CMA-ES again
4. With acceptable error, use Simplex to get a more precise result
5. I also used a Multi-Start-Multivariate optimizer with CMA-ES and simplex to help avoiding local optima

This gave acceptable results