Let $f:\mathbb{R}^2\to \mathbb{R}$ be defined by $f(x, y)= x^6-2x^2y-x^4y+2y^2$. Then I need to show that $(0, 0)$ is a saddle point of $f$.
My effort: We have $f_{xx}= 30x^4-4y-12x^2y$, $f_{xy}= -4x-4x^3$, $f_{yy}= 4$. At $(0, 0)$, we have $$f_{xy}^2-4f_{xx}f_{yy}= 0$$ at $(0, 0)$. Also $f_{xx}=0$, means that second order derivative test is inconclusive. So, further test is required.
Observe that $f(h, k)-f(0, 0) = h^6-2h^2k-h^4k+2k^2$. For $h = k$, we have $$f(h, h)-f(0, 0)= h^6-2h^3-h^5+2h^2$$ As $h$ is near $0$, $$f(h, h)-f(0, 0)>0.$$ For $h \neq k$, $$f(h, k)-f(0, 0) = h^6-2h^2k-h^4k+2k^2 = h^2(h^4-2k-h^2k)+2k^2$$
How to proceed further? Am i proceeding in right direction or not?
We have that
$$f(x,cx^2)=(1-c)x^4(x^2-2c).$$ Thus
$$0<c<1, 0<|x|<\sqrt{2c}\implies f(x,cx^2)<0.$$
On the other hand $f(x,0)=x^6.$