$x^2y \rightarrow$ max,
such that $x^2 + 4xy \leq 1, x \geq 0$ and $y \geq 0$.
I think I need to use the KKT conditions here. I did however not yet succeed in solving it, so could someone please give me an example of how this should be done? And should I include the constraints $x \geq 0$ and $y \geq 0$ into the Lagrangian function?
The KKT conditions in this case for $x,y$ that are feasible ($x\geq0,y\geq0,x^2+4xy\leq1$) is that $\exists \alpha,\beta,\gamma\geq0$ (feasible dual variables for the constraints in the last parens in respective order) such that
However, the problem is not convex therefore KKT conditions are not sufficient. In particular, the eigenvalues of the hessian of your objective are $y\pm\sqrt{4x^2+y^2}$ so that one of them is always positive (so not concave for a maximization problem) and also one is negative whenever $x>0$ which is certainly feasible (so it's not even definite in either direction).
But they are necessary. So you can guess which constraints are active or not and go through all permutations of this to find all KKT-satisfying points. For example, guess that the nonnegativity constraints are slack ($\alpha=\beta=0$) and that the other constraint is tight ($x^2+4xy=1$). Then plugging this into the KKT conditions we get three equations: $$2xy=\gamma(2x+4y)$$ $$(1-\gamma)x^2=0$$ $$x^2+4xy=1$$ Which solve for $x=1/\sqrt{3}$ and $y=1/\sqrt{12}$ and give objective value $1/\sqrt{108}$. (Hint: this is optimal.)