What is the difference between the method of lagrange multipliers and penalty-based methods?

242 Views Asked by At

Both the method of Lagrange multipliers and penalty-based methods involve converting a constrained optimization problem into an unconstrained one by creating a new function $\mathcal{L}$ that penalizes $\mathcal{L}$ when the constraint is violated. In both of these methods, the general form of $\mathcal{L}$ is:

$$\mathcal{L} (\mathbf{x}, \lambda) = f (\mathbf{x}) + \lambda \cdot \varphi (h (\mathbf{x}))$$

where $f (\mathbf{x})$ is the objective function, $\varphi (h (\mathbf{x}))$ is a penalty function of the constraint, and $\lambda$ is some parameter.

In the case of the method of lagrange multipliers, $\varphi (h (\mathbf{x})) = h (\mathbf{x})$, whereas in penalty-based methods, $\varphi$ is usually chosen to be the square function. Aside from the choice of $\varphi$, I don't see the difference between the two methods.

1

There are 1 best solutions below

0
On

Penalty methods are algorithmic. Given a suitable $\lambda_k$, our aim is to minimize the objective function $ \mathbf{x}_k= \arg\min f (\mathbf{x}) + \lambda_k \cdot \varphi (h (\mathbf{x}))$. Eventually, the sequence $\{\lambda_k, \mathbf{x}_k\}$ converges to a local minimum. The Lagrange multipliers method is global, and under some regularity conditions involving $ \{f (\mathbf{x}), h (\mathbf{x})\}$, we can determine all stationary points, that need further qualification.