How to understand the improvement of Levenberg-Marquardt algorithm to Gauss-Newton method?

Question

I have read wikipedia about Levenberg-Marquardt algorithm. The L-M algorithm introduces a damping parameter $\lambda$ to adjust the step at each iteration. I want to know : what is the disadvantages of Gauss-Newton method that L-M want to overcome ? And why L-M algorithm is sensitive to the initial position ?

Answer 1

GN can behave rather wildly far away from a local minimum. In particular, where gradient descent (GD) will always improve the objective function, GN may not.

LM amounts to hybridizing GN with GD. In particular, as the damping parameter $\lambda$ is increased, the direction that you take becomes closer to the GD direction.

LM with small $\lambda$ has the same unpredictable behavior as GN. LM with large $\lambda$ has the same slow movement and vulnerability to local minima as GD.

LM also has the advantage that its linear systems have smaller condition numbers than those for GN.