My background is not mathematics, i always face it hard to work with different constrained optimization problem though i have to.
Normally what i understand is, when i have a optimitization problem with some constraints i can use lagrange multiplier to make it to unconstrained optimization problem and solve it.
But recently i have come up with a paper where this sort of problem was being approached with an augmented lagrangian. Now, what it seems to me is that augmented lagrangian method is kind of a variant for what is called the 'penalty method'. But i was wondering when to use the lagrange and when we should use augmented lagrangian?
Can anyone explain the situation in simple words where to apply what? Many thanks.
For solving the optimization problem with linear equality constraints, I think the Lagrange Multiplier has no advantage over Augmented Lagrangian.
Simply put, Lagrange Multiplier can be seen as linear penalty, and Augmented Lagrangian can be seen as quadratic + linear penalty. Since Augmented Lagrangian has more ``penalty'', the equality constraints are easier to be satisfied, which means that Augmented Lagrangian will converge faster.
Besides, in M.J.D.Powell's paper, we can know that the Augmented Lagrangian can also be seen as a kind of approximation of Newton method.