My work. $$f(x,y) = 2x^4 - 3x^2y + y^2.$$ We differentiate partially and find $$f_x = 0$$ and $$f_y=0,$$ which is indeed $0$ at $(0,0)$. Now $f_{xx} = 0$, $f_{yy}=2$ and $f_{xy}=0$. Since $f_{xx}*f_{yy} - f^2_{xy} = 0$, this requires a further probe. So I calculated $d^2f = h^2f_{xx} + 2hkf_{xy} + k^2f_{yy} = 2k^2 $ which keeps the same sign for all values of $(h,k) \neq (0,0)$. But in the solution it says $(0,0)$ is neither a maxima nor minima as $f(x,y)-f(0,0)$ doesn't keep the same sign at the origin. My doubt is the even though necessary and sufficient conditions for the extremum are fulfilled then why can't we say the function has extremum. I have a conflict in the definition that $f(x,y)$ has to keep same sign near the extremum. Can anyone help where I am going wrong in the process. P.S. The book says the following.
$f(a,b)$ is extreme value of $f(x,y)$ if at $(a,b)$, $df = 0$ and $d^2f$ keeps the same sign for all values of $(h,k) \neq (0,0)$.
When they say "keep the same sign" it means strictly positive or strictly negative. In your case, if you choose $h=1, k=0$ for example, you have $d^2f(1,0)=0$ so more information is needed to draw a conclusion. The condition $(h,k)\neq (0,0)$ means not both $h$ and $k$ is equal to zero (the vector $(h,k)$ is not the null vector).
To understand what happens at $(0,0)$, notice that $$ f(x,y)=(x^2-y)(2x^2-y). $$ Therefore, in the region where $x^2<y<2x^2$, which contains points arbitrarily close to the origin, you have $f(x,y)<0$. On the other hand, along the $X$-axis $y=0$, $x\neq 0$, you have $f(x,0)=2x^4>0$. Therefore your point is neither a local maximum or a local minimum.