Investigate maxima and minima of $$f(x,y)=x^2-2xy+y^2+x^3-y^3+x^5$$
My attempt:
$$f_x=2x-2y+3x^3+5x^4=0$$ $$f_y=-2x+2y-3y^2=0$$ Solving,I got $(0,0)$ as a stationary point.
The second derivative test fails in this case
The solution is $(0,0)$ is a saddle point but I can't figure out how
Let us construct the Hessian $$H(x,y) = \left[\begin{array}{cc} f_{xx} & f_{xy} \\ f_{yx} & f_{yy} \end{array}\right]$$ $$= \left[\begin{array}{cc} 2+6x+20x^3 & -2 \\ -2 & 2-6y \end{array}\right] $$ $$\implies |H|(0,0) = \left|\begin{array}{cc} 2 & -2 \\ -2 & 2 \end{array}\right| = 0 $$ Hence, we must proceed with definition of maxima/minima saddle point. At the point $(0,0)$, we see function value is $0$. Now in an $\epsilon$-neighbourhood of the point, if the function values are all bigger then maxima, else if all smaller then minima. If there are some above and below, no matter how small epsilon, then it's a saddle point. Just look at $f(\epsilon,\epsilon) = \epsilon^5$, this will change sign about $(0,0)$, hence saddle point