I had the question $$x^2+y^2+xy=1$$ and I had to find maximum value of $$\vert xy(x^2+y^2) \vert$$
I tried using Lagrange multipliers by forming
$x^3+3xy^2 = \lambda (2y+x)$ And $y^3+3yx^2 = \lambda (2x+y)$
Now I subtract it to get $(x-y)^3 = \lambda (y-x)$
And as $\lambda ≠ 0 , x=y.$ This gives$ x = {1 \over \sqrt{3}}$ and the expression gives max as $\frac{2}{9}$ but the answer is $2$ . What have I done wrong.
I know it's modulus , but with negative too, there will only be a change in the sign of $\lambda$ which will not affect any working.
Also Lagrange Multipliers is something I read as an extra piece of info online, so maybe I might be missing something trivial here.
Also is there any requirement on $\lambda$ to be positive ??

Find the maximum $M$ and the minimum $m$ of $xy(x^2+y^2)$ on that region. Then:
Concerning your approach, it seems that you tried to find the maximum of $\bigl\lvert xy(x^2+y^2)\bigr\rvert^2$, which is equal to $x^2y^2(x^2+y^2)^2$. It works, but then the system that you will have to solve will be$$\left\{\begin{array}{l}2xy^2\left(3x^4+4x^2y^2+y^4\right)=\lambda(2x+y)\\2xy^2\left(3x^4+4x^2y^2+y^4\right)=\lambda(x+2y)\\x^2+y^2+xy=1,\end{array}\right.$$instead of the one that you mentioned. It can be done, but it leads to tougher computations.