Lagrange Multiplier Constraint

Question

If I have a function $f(x,y)$ subject to the constraint $g(x,y)=c$ At constrained min/max, in any direction along level $g=c$ the rate of change of $f$ must be zero. As per the professor, in the unconstrained case, the partial derivatives must be zero only in certain allowed directions. But when there is a constraint the derivatives are zero only in an allowed direction. What does this mean?

Answer 1

I think you've misquoted your professor. If $f(x,y)$ has an unconstrained extrema $(x_0,y_0)$, then $\nabla f(x_0,y_0) = 0$. The direction of approach towards $(x_0,y_0)$ doesn't matter. All directions are allowed in the unconstrained case. If $f$ is constrained by $g(x,y)=c$, then $\nabla f$ need never be $0$ on $g^{-1}(c)$. The only condition required is that $\nabla f$ is some scalar multiple of $\nabla g$. once again, direction of approach doesn't matter. Now when does the direction of approach matter? It's when you're looking at the parameterized function $f(r(t))$ for some parameterization $r(t) \in g^{-1}(c)$. Now the derivative of the parameterized function with respect to $t$ does disappear at an extreme value of $f$ subject to $g=c$ i.e $\frac{d}{dt} f(r(t)) = \nabla f \cdot r'(t) = 0$. Now the direction of approach matters. Because notice that the parameterization $r(t)$ lives in the set $g^{-1}(c)$, so $r'(t)$ has to be tangent to the surface $g=c$.

I'm assuming this sounds a bit complicated so I'll demonstrate with an example

Suppose you're function $f(x,y) = x^2-4x + y^2 + 4$ and your constraint is the unit circle $g(x,y) = x^2+y^2 = 1$.
Unconstrained Case: Extrema of $f$ occurs at $$\nabla f = 0 \implies \begin{bmatrix}x \\ y\end{bmatrix} = \begin{bmatrix}2 \\ 0\end{bmatrix}$$ Now because $\nabla f = 0$, $\nabla f \cdot \vec{v} = 0$ for every vector $\vec{v}$, so no matter which direction you're approaching this extreme value, the gradient is always $0$.
Constrained Case: First we can note that $\nabla f$ should be a scalar multiple of $\nabla g$ (think about why this is true). So $$\nabla f = \lambda \nabla g\,; \quad g=c$$ gives us solutions: $(x,y,\lambda) = (-1, 0, 3), (1, 0, -1)$. Let's check the value of $\nabla f$ at these points, first at $(x,y) = (-1,0)$, we have:

$$\nabla f = \left[\begin{matrix}-6\\0\end{matrix}\right]$$

Now it's easy to see that $\nabla f \cdot \vec{v} = 0$ for some but not all $\vec{v}$. Further analysis would show you're looking for those $\vec{v}$ that are tangent to the unit circle at $(-1,0)$. Similarly, if we check $(x,y) = (1,0)$, we get

$$\nabla f = \left[\begin{matrix}-2\\0\end{matrix}\right]$$ Once again $\nabla f \cdot \vec{v}$ disappears only when $\vec{v}$ is tangent to the constraint at $(1,0)$

Answer 2

If a local extrema is inside your domain, then the gradient in any direction must be zero. If the gradient in a direction is non-zero, you can move your evaluation point in that direction and you will obtain higher / lower value (depends on if the gradient is positive or negative in that direction), thus your point is not a local extrema.

If a local extrema is at the boundary of your domain, then the gradient in the direction away from your domain need not be zero, since you can't move away from your domain anyway.

Another possible location of the local extrema is at points in which the gradient does not exist.