I am trying to understand the relationships in the KKT theorem between being a maximizer, satisfying the first order conditions (FOCs) and complementary slackness (CSC), and the linearly independent constraint qualification (LICQ). My understanding is given schematically by the following Venn diagram. Is this correct?
It would also be great to know how having a concave Lagrangian (which, along with the FOCs+CSC, gives a sufficient condition for a maximizer) fits in this diagram.
Any insight is appreciated!
Note: By FOCs, I mean the stationarity condition, and I include primal/dual feasibility conditions in my use of "CSC."
Let's have the ideas clear. First, take a special care of how the conditions are being called. In the literature on nonlinear programming, It seems not pretty well-established how the concept of a first-order condition (FOC) is defined in your context. Sometimes it means the KKT conditions itself and at another moment it is the "KKT or not-CQ" optimality condition. Both notions are different in the context of nonlinear programming. I will suppose that by first-order conditions (FOCs) and complementary slackness (CSC) you directly mean the KKT conditions. Some classic books call the KKT conditions a first-order optimality condition, like the one by Luenberger. Second, your diagram is not clear if it does not follow a caption. I will suppose that by your diagram you mean that, for a fixed and possible nonlinear constrained optimization problem, the Venn diagram represents the feasible points in that problem so that they satisfy the conditions you put in the picture. Third, I will adopt the KKT conditions for minimization, and not maximization, without any loss of generality, but just because I'm used to it. In such a context of the three observations, I can clarify your question precisely without any confusion.
In the nonlinear programming context, and under the three observations above, your diagram completely makes sense and has nothing wrong with it. Just to point it out, the points in which LICQ holds have nothing to do with minimizers, and that's why its associated ellipse might not intercept the minimizers in any particular way. Additionally, the FOCs+CSC conditions, a.k.a. KKT conditions, can only be guaranteed when a constraint qualification holds. In your case, when LICQ holds, we can assure the KKT conditions for minimizers. And lastly, when the KKT conditions hold at a point, not necessarily such a point is one of its minimizers. Here, the KKT conditions are just a sense of stationarity.
In the context of a convex Lagrangian, what will happen is not too different except by the last fact: $$\text{when the KKT conditions hold at a point, necessarily such a point is one of its minimizers}.$$ This is true due to the fact that when the KKT conditions hold at $\boldsymbol{x}$ for some multipliers $\boldsymbol{\mu} \in \mathbb{R}^{p}_{+}$, $\boldsymbol{\lambda} \in \mathbb{R}^{q}$ (i.e., it holds \begin{equation*} \begin{array}{c} \nabla f(\boldsymbol{x}) + \sum^{p}_{i=1} \mu_i \nabla g_i (\boldsymbol{x}) + \sum^{q}_{j=1} \lambda_j \nabla h_j (\boldsymbol{x}) = \boldsymbol{0}\\ g_i (\boldsymbol{x}) \mu_i = 0, \forall\ 1 \leq i \leq p) \end{array} \end{equation*} in which the Lagrangian in the primal variable (i.e., $\boldsymbol{x}$) is convex, it holds, for feasible points $\boldsymbol{x}+\boldsymbol{h}$, \begin{align} f(\boldsymbol{x}+\boldsymbol{h}) \geq & f(\boldsymbol{x}+\boldsymbol{h}) + \sum^{p}_{i=1} \mu_i g_i (\boldsymbol{x}+\boldsymbol{h}) + \sum^{q}_{j=1} \lambda_j h_j (\boldsymbol{x}+\boldsymbol{h}) \\ \geq & f(\boldsymbol{x}) + \sum^{p}_{i=1} \mu_i g_i (\boldsymbol{x}) + \sum^{q}_{j=1} \lambda_j h_j (\boldsymbol{x}) +\\ & \left( \nabla f(\boldsymbol{x}) + \sum^{p}_{i=1} \mu_i \nabla g_i (\boldsymbol{x}) + \sum^{q}_{j=1} \lambda_j \nabla h_j (\boldsymbol{x}) \right)^{T} \boldsymbol{h}\\ \geq & f(\boldsymbol{x}). \end{align} Hence, the ellipse of the KKT points need to be inside of the minimizers one. In such a context, the similar Venn diagram takes the form: