Gradient´s direction of constraints (Lagrange multiplier)

Question

I am trying to understand Lagrange´s multiplier method. I have one question. I´ ve seen sketches where the gradient of function f and constraint g point to the same direction at a possible extreme value. However, there are also sketches where g´s gradient points to the opposite direction of f´s gradient at the possible extreme value. Is there any meaningful interpretation of the direction of the constraint´s gradient?

Answer 1

No, there is not. Suppose that you have two constraints $g_1$ and $g_2$, with $g_2=-g_1$. You want to find the local extremes of the restriction of $f$ to the set $\{x\in\Bbb R^n\mid g_1(x)=c\}$, for some $c\in\Bbb R$, which is equal to the set $\{x\in\Bbb R^n\mid g_2(x)=-c\}$. If $p$ is such that $\nabla f(p)=\lambda\nabla g_1(p)$ for some $\lambda\in\Bbb R$, then $\nabla f(p)=-\lambda\nabla g_2(p)$. So, either $\nabla f(p)$ and $\nabla g_1(p)$ have the same direction or $\nabla f(p)$ and $\nabla g_2(p)$ have the same direction. But, again, its the same set. So, having the same direction or opposite directions means nothing.

Answer 2

The basic idea of this method is that in the point $p$ of possible constrained extremum a derivative of $f$ along any curve passing through $p$ is zero at $p$. Hence the gradient of $f$ belongs to the orthogonal complement of a tangent space to $g$ in $p$. Hence it can be represented as a linear combination of gradients of functions defining the constraints.

Answer 3

To elaborate on my comment (everything taken with a grain of salt since I'm just studying this subject myself):

Let $$\Omega=\{x\in\mathbb R^n : g(x)\geq 0, h(x) = 0\}$$ be the admissible set for continuously differentiable constraints $g: \mathbb R^n \to \mathbb R^m$ and $h: \mathbb R^n \to \mathbb R^k$.

Let $y\in\Omega$ and $g_i: \mathbb R^n \to \mathbb R$ be active in $y$, i.e. $g_i(y)=0$, and $\nabla g_i(y)\neq0$. Then there exists a neighbourhood of $y$ such that the set $M=\{x\in\mathbb R^n:g_i(x)=0\}$ is a $C^1$-submanifold of $\mathbb R^n$. In $y$, the vector $\nabla g_i(y)$ will be orthogonal to $M$ and pointing in the direction of increasing values of $g_i$, that is into the set $\{x\in\mathbb R^n: g_i(x)\geq0\}$.

As an example take the set above the parabola: $$\Omega=\{x\in\mathbb R^2: g(x) \geq 0\}$$ where $g(x)=x_2-x_1^2$.

We have $$\nabla g(x)=\begin{pmatrix} -2 x_1 \\ 1 \end{pmatrix}.$$

In the origin the gradient is $\begin{pmatrix} 0 \\ 1 \end{pmatrix}$, pointing up vertically as expected. In $\begin{pmatrix} 1 \\ 1 \end{pmatrix}$ it is $\begin{pmatrix} -2 \\ 1 \end{pmatrix}$, in $\begin{pmatrix} 2 \\ 4 \end{pmatrix}$ it is $\begin{pmatrix} -4 \\ 1 \end{pmatrix}$ and so on. The further we travel on the parabola the more horizontal the gradient becomes, always pointing into $\Omega$ orthogonal to its boundary.