Suppose that $V$ is open in $\Bbb R^2$ that $(a,b)\in V$$\ \ \ \ \ f:V\to\Bbb R$ has second order partial total differential on $V$ with $f_x(a,b)=f_y(a,b)=0$
If the second order partial derivatives of f are continuous and exactly two of three numbers $f_{xx}(a,b)$ $f_{xy}(a,b)$ and $f_{yy}(a,b)$ are zero,
Then prove that (a,b) is saddle point if $f_{xy}(a,b)\not= 0$
I think that
Since $f_{xy}(a,b)\not=0$ $$f_{xx}(a,b)=f_{yy}(a,b)=0$$
I want to solve this question by using the theorem.
Thm: let $A,B,C\in \Bbb R$ and $D=AC-B^2$ and $\varphi $(h,k)=$Ah^{2}+2Bhk+Ck^2$
if
(i) $D>0$ and A, $\varphi(h,k)$ have same sign $\forall (h,k)\not= 0$
(ii) ıf $D<0$ then $\varphi (h,k)$ takes on both positive and negative valuesas (h,k) varies over $\Bbb R^2$
But, after here, I dont have any idea. Show me the solution? Thank you.
Just take alook at eigenvalues of the hessian (let $s=f_{xy}(a,b)$), which has the form $$\begin{pmatrix}0&s\\s&0\end{pmatrix}$$with $s\ne 0$. It has eigenvalues $\pm s$, so they are of different sign, hence this point is a saddle.
Also, you might want to take a look at wiki article.
Edit
The theorem you want to use is, actually, studying the bilinear form defined by the matrix $$M=\begin{pmatrix}A&B\\B&C\end{pmatrix}.$$ If you take a vector $x=\begin{pmatrix}h\\k\end{pmatrix}$ and examine$$x^TMx,$$ you obtain your function $\varphi$.
1) $\varphi$ is always positive for $x\ne 0$ $\iff$ all eigenvalues of $M$ are positive.
2) $\varphi$ is always negative for $x\ne 0$ $\iff$ all eigenvalues of $M$ are negative.
3) $\varphi$ can be both positive and negative $\iff$ one eigenvalue is positive and another is neagtive.
Note that the position of eigenvalues is decribed in terms of $A$, $B$, and $C$.
In your particular case, look at $x_1= \begin{pmatrix}1\\1\end{pmatrix}$, $x_2= \begin{pmatrix}1\\-1\end{pmatrix}$; $\varphi(x_1) = 2s$, $\varphi(x_2) = -2s$. Clearly, $\varphi$ can be both positive and negative, hence $(a,b)$ is a saddle point.