how to use a graph to understand constraint optimization and Lagrange multipliers

669 Views Asked by At

enter image description here

let's say we have a circle on x-y plane and then we tilt the circle around 30 degree and point A is anchor. enter image description here

we want to maximize the distance from the point $(x, y, z)$ to the origin $$f(x,y,z)=x^2+y^2+z^2$$ subject to the constraint that the point lie on the circle. It is obvious that B is the point we want. But according to the optimisation theory, $\nabla f(x,y,z)$ has to be orthogonal to the tangent plane of B in the circle, but that's impossibile, I know I must have something wrong, but where do I go wrong?

1

There are 1 best solutions below

6
On BEST ANSWER

You're thinking in terms of, for example, optimizing a function of two variables on a curve in $\Bbb{R}^2$, or optimizing a function of three variables over a surface in $\Bbb{R}^3$. In these situations, you have a function $f$ which you want to optimize, and a function $g$ so that $g(\mathbf{x})=c$ defines your constraint. Then (roughly) at a local maximum or minimum, we have $\nabla f = \lambda \nabla g$, where $\lambda$ is a constant, i.e. $\nabla f$ is parallel to $\nabla g$, causing $\nabla f$ to be perpendicular to the curve.

However, this situation is different; we're looking to optimize a function of three variables over a curve in $\Bbb{R}^3$. Provided that $g$ is "nice" enough to work with for these purposes at $c$, any constraint $g(\mathbf{x}) = c$ would actually define a surface (intuitively: $g$ is a function of three variables, setting $g(\mathbf{x})=c$ removes one degree of freedom, leaving a region parametrized by two variables). So, in order to describe the curve in the OP, we wold need two constraints: $g_1(\mathbf{x}) = c_1$ and $g_2(\mathbf{x})=c_2$. In this situation, you find the local extrema by solving the following system of equations: $$\nabla f = \lambda \nabla g_1+\mu \nabla g_2$$ Where $\lambda, \mu$ are constants. This equation does not impose any impossible constraints on the direction of $\nabla f$. Geometrically speaking, the curve is the intersection of two surfaces given by $g_1(\mathbf{x}) = c_1$ and $g_2(\mathbf{x})=c_2$ (one for each constraint), and at $B$, $\nabla f$ must merely be in the plane generated by $\nabla g_1 (B)$ and $\nabla g_2(B)$.