$f(x,y) = (x^2 +y^2)^2 - 2x^2 + 2y^2$ So
$ \frac{∂}{∂x} f(x,y)=4x(x^2+y^2)-4x $
$ \frac{∂}{∂y} f(x,y)=4y(x^2+y^2)+4y $
After i tried to find the critical point :
$ 4x(x^2+y^2-1)=0 $
$ 4y(x^2+y^2+1)=0 $
I'm stuck at this point , i can't find the critical point and therefore find the hessian
From the second equation, $y=0$. That's because $$x^2+y^2+1\ge 1$$for any real $x$ and $y$. So than either $x=0$ or $x^2-1=0$. So the critical points are $(0,0)$, $(-1,0)$, and $(1,0)$.