I am currently reading the book An introduction to optimization by Chang and Zak.
When reading about the Karush Kuhn Tucker (KKT) conditions, I came across this geometrical explanation of the KKT theorem at page 489.
The book then states that $g_j(x) \le 0,\, \, \, j=1,2,3$ and that $x^*$ is a minimizer. Also $g_3(x) \le 0$ is inactive: $g_3(x) \lt 0$. My question is, why is $g_3(x)$ inactive (i.e how can one see this from Figure 21.1) and why are the negative gradients of the other two constraints poininting in those directions?
Thank you.
Inactive Constraint
A constraint $g(x) \le 0$ being inactive at a feasible point $x$ means that $g(x) < 0$. In your picture $x^*$ is feasible but it is not on the level curve of $g_3(x) = 0$, so it means that $g_3(x^*)< 0$, i.e. $g_3$ is inactive at $x^*$.
Gradients
The exact direction that the gradients are pointing is not so important. What is important is the following:
You see the little dashed lines by $x^*$? Those are supposed to represent that this picture is potentially taking place in a higher dimensional space (like $2, 3, 4, ...$). So you have the two random vectors $-\nabla g_1(x^*)$ and $-\nabla g_2(x^*)$ in a (let's say) $3$ dimensional space. What's the chance that a third random vector, $\nabla f(x^*)$, will lie in the plane spanned by the first two? Seems like not very high. But that's precisely what the KKT theorem says. It says that at the minimizer $x^*$, the gradient $\nabla f$ will lie in the plane spanned by the gradients of the (active) constraints.