Lagrange Multiplier do not make sense

Question

Basically the core of Lagrange's multiplier says that the solution to a constrained optimization occurs when the contour line of the function being maximized/minimized is tangential to the constraint curve. I am not able to convince myself of above statement. Consider below diagram wherein 'f' is some function being maximized/minimized and g(x,y) is the constraint. the contour lines of f and g(x,y) are plotted.

Clearly the contour line which corresponds to local maximum of f is not tangential to g i.e. the gradient vector of f and g don't align at the intersections of f=5 and g. I am not looking for a rigorous proof or something, just want to get a basic understanding of why gradient of f and g should point in same direction at the local maxima/minima of f?

Answer 1

Think of the function to be minimized as the distance to a given point, subject to given constraints. From any point, (x,y,z), if there were no constraint the obvious thing to do would be to move directly toward that given point, along the vector from (x,y,z) to the given "target" point. But with the constraint, we cannot do that- we have to stay on a curve satisfying that constraint. What we could do is look at the projection of the vector from (x,y,z) to the target point on the curve. If that projection is to the right, we move right on the constraint curve. If that projection is to the left, we move left on the constraint curve. We keep doing that until we cannot do it: until the vector from (x,y,z) to the target point does NOT have a projection to the right or the left- until it is perpendicular to the constraint curve.

And that is exactly what the "Lagrange multiplier" method does! $\nabla f$ points directly from (x,y,z) to a local max or min. And $\nabla$ of the constraint function is always perpendicular to a constraint curve. Saying that one is $\lambda$ times the other says they point in the same direction.

Answer 2

The intuition that people get from the Lagrange multiplier method, namely that $\nabla f = \lambda\nabla g$ must mean that the contour lines are tangent, is not always correct. Your example, or taking $f(x,y)=x^2, g(x,y)=x^2+y^2-1$, shows this. Note however this does not mean that the Lagrange method does not work, simply that the intuition of tangency is not always correct.

Answer 3

What Lagrange multiplier says is somewhat more general, it says that in a local maximum the gradient of the function being optimized is collinear with the gradient of the function defining the constraint.

And, if neither of the gradients vanishes then this is indeed equivalent to your formulation: tangentiality of the contours. But, if the gradient of the function is zero somewhere then Lagrange criterion is also satisfied in that point (we consider the $0$ vector collinear with any vector, as it is a multiple of any vector).

If the function $f$ described by your contour lines is differentiable then its gradient must be zero somewhere on that circle. Say, if $5$ is its actual maximum then $\text{grad } f$ vanishes along the whole $f=5$ line.