I try to compute extremum points for this funciton: $$ f(x,y) = x^{4} - y^{4} - 4xy^{2}-2x^{2} $$ The first step compute gradient: $$ \nabla f(x,y) = [4 x^3-4 x-4 y^2, -8 x y-4 y^3] $$ Next step $$ \nabla f(x,y) = 0 $$ and its fail. It's not easy compute explicitly roots.
Do you have any ideas?
Your system is$$\left\{\begin{array}{l}x^3-x-y^2=0\\2xy+y^3=0.\end{array}\right.$$If $y=0$, then the second equation becomes $0=0$, whereas the first one becomes $x^3-x=0$. So, the solutions for which $y=0$ are $(\pm1,0)$ and $(0,0)$.
If $y\ne 0$, then the second equation is equivalent to $2x+y^2=0$, that is, $y^2=-2x$. But then the first equation becomes $x^3-x+2x=0$, or $x^3+x=0$. Can you take it from here?