Let $f:\mathbb{R}^2 \rightarrow \mathbb{R}$ such that $f(x,y) = x^4 + y^4 - 2(x-y)^2$. Study its extrema.
So here was my approach.
We have $$\frac{\partial f}{\partial x}(x,y) = 4(x^3 -x + y),\frac{\partial f}{\partial y}(x,y)= 4(y^3 -y + x) $$ I have to find $(x_0,y_0) \in \mathbb{R}^2$ such that $ \frac{\partial f}{\partial x}(x_0,y_0)= \frac{\partial f}{\partial y}(x_0,y_0) = 0 $
So we have:
$$\left\{\begin{matrix} x(x^2-1)+ y =0\\ y(y^2 -1) + x =0 \end{matrix}\right. $$ Thus we have $x-x^3 = y$, and by replacing $y$ with $x-x^3$ in the second line, we get:
$$ y(y^2 -1) + x =0 = (x-x^3)((x-x^3)^2 -1)+x =x^5(-x^4 -x^2 -3) = 0 $$ And the only solution for this is $x=0$. As $y = x^3 -x$ we immediately have $y=0$.
So the only extremum possible is at $(0,0)$. Now, I need to study its Hessian matrix.
We have: $$\frac{\partial ^2f}{\partial x^2}(x,y) = 12x^2 -4 , \frac{\partial ^2f}{\partial y^2}(x,y) = 12y^2 -4, \frac{\partial^2 f}{\partial x \partial y}(x,y) = \frac{\partial^2 f}{\partial y \partial x}(x,y) = 4$$
Thus $$H(x,y) \begin{bmatrix} 12x^2 -4&4 \\ 4& 12y^2 -4 \end{bmatrix} $$
At $(x,y)=(0,0)$ we have $$H(0,0) \begin{bmatrix} -4&4 \\ 4& -4 \end{bmatrix} $$
As we have $\text{det}(H(0,0)) = 0$ and $\text{Tr}(H(0,0)) = -8$ the eigenvalues are $0$ and $-8$. As it has $0$ as eigenvalue, I need to study the differential at a higher order. But here I lack understanding. What exactly should I do? Should I compute the third order partial differentials and then restudy their Hessian Matrix?
The matrix $H(0,0)$ is negative semidefinite since
$$\left\langle \begin{bmatrix}-4 & 4 \\ 4 & -4\end{bmatrix}\begin{bmatrix}x \\ y\end{bmatrix}, \begin{bmatrix}x \\ y\end{bmatrix}\right\rangle = -4(x-y)^2 \le 0$$
with equality iff $x = y$.
This is a necessary condition for a local maximum, but not sufficient. Therefore, the test is inconclusive.
Indeed, $(0,0)$ is a saddle point. If we approach $(0,0)$ along $x = y$ then we have
$$f(x,x) = 2x^4 > 0$$
However, if we approach along $x = -y$, then
$$f(x,-x) = 2x^4-8x^2 < 0$$
near $(0,0)$.