What does the Lagrange multiplier of an equality constraint mean, intuitively?

1.6k Views Asked by At

Consider a nonlinear optimization problem of the form \begin{align} \min_{x}&\quad f(x)\\ \nonumber \text{subject to } \quad&h_i(x) = 0,\,i=1,\ldots,I\\ \nonumber \quad&g_j(x) \le 0,\,j=1,\ldots,J\\ \nonumber \quad&x\in X\subseteq\mathbb{R}^n \end{align}

I understand the intuitive meaning of the dual variable associated with an inequality constraint, $h_i(x)\le0$ -- it can be interpreted as the change in the objective value when we relax the constraint by a small amount (say change $0$ to some $\alpha>0$).

On the other hand, the dual variables for equality constraints are, in general, unrestricted in sign. Can someone provide some intuition for what the dual variables of an equality constraint mean?

1

There are 1 best solutions below

0
On BEST ANSWER

Here's the way I think of it: in unconstrained optimization, you can think of $-\nabla f$ as a "force" that pulls any point in $\mathbb{R}^n$ towards the local minimum. You are at the minimum when $\nabla f =0$.

Now suppose you have some constraints (equality or inequality). The point feels a force in the $-\nabla f$ direction, but is constrained to move tangentially to the level sets of any active constraints at the current point. The point is at a minimum when $\nabla f$ is in the span of the $\nabla h$ and the $\nabla g$ for the active inequality constraints.

The $\lambda \nabla h$ can be thought of as a set of forces just strong enough to exactly counteract the components of $-\nabla f$ that would push the point into the inadmissible region. If the constraint gradients are orthogonal then $\lambda \nabla h$ is the infinitesimal change in the value of the constraint $h(x)$ due to infinitesimally moving $x$ in the $-\nabla f$ direction. The sign of $\lambda$ tells you whether this motion would take the point into the region $h<0$ or $h>0$, i.e. whether the constraint is "pushing" or "pulling" the point to keep it on $h=0$. (Inequality constraints can only push which is why their Lagrange multipliers are always positive.) In general this interpretation is not quite accurate, since the $\nabla h$ are not usually orthogonal to each other: the force from constraint $h_1$ can help or hinder keeping $x$ out of $h_2$'s inadmissible region.