Let $$f(x,y)=xy^2+3x^2y-2xy\quad{ et }\quad g(x,y)=x^2+y^2-1.$$ I want to find the maximum and minumum value of $f$ under the constraint $g(x,y)=0$ by the method of Lagrange multipliers.
The Lagrange multipliers function is $$L(x,y,\lambda)=xy^2+3x^2y-2xy+\lambda(x^2+y^2-1).$$ $(x,y,\lambda)$ is a critical point of $L$ if and only if $$ \begin{cases} y^2+6xy-2y+2\lambda x=0\\ 2xy+3x^2-2x+2\lambda y=0\\ x^2+y^2-1=0 \end{cases} $$
I hope to find the critical points of $f$.
My question is taken from en exercice of an exam
"Abandon hope all ye who enter here" as wrote Dante Alighieri in "The Divine Comedy".
Either there is a typo or the problem needs to be solved numerically.
The only thing you can do is to extract $x$ from the first equation or $y$ from the second and plug it in the remaining equations. I suppose that you see the monsters you would need to face.
Just for your curiosity, $x$ is given by one of the six real solutions of $$90 t^6-72 t^5-107 t^4+84 t^3+27 t^2-24 t+3=0$$ and this one is the simplest.
If you want to visualize the problem, eliminate $x$ from the first equation, replace it in the second and third and look at the contour plot of $$\Phi=\Big[2xy+3x^2-2x+2\lambda y]^2+\Big[x^2+y^2-1\Big]^2 \quad \text{where} \quad x=\frac{y(2-y)}{2 (3y+\lambda )}$$ Select the range $-2 \le y \le 2$ and $-4 \le \lambda \le 4$; draw the level curve corresponding to $\Phi=0.5$ to visualize the six areas.
More awful is the $3D$ plot of $\Phi$.