The question asks to "show that the condition for $P(x,y)=ax^2+2hxy+by^2$ ($a$,$b$ and $h$ not all zero) to be positive definite is that $h^2<ab$, and that $P(x,y)$ has the same sign as $a$."
Now I've seen questions similar to this before where it's a two variable quadratic and I'm not too sure how to go about it. Normally with one variable you could just show the discriminant is less than $0$ but since there's two variables I can't use the same process (since I'm not sure how the discriminant is defined for a two variable quadratic). I also tried completing the square but that doesn't seem to make the algebra anything near as simple as the answer (I got a rather complicated fraction with cubes). Is there a way to go about this type of problem? Thanks.
In higher dimensions (than 1), there is a elegant matrix based approach. In this approach, the sign is determined by the diagonal matrix. Specifically, note that for your question $$a{x^2} + 2hxy + b{y^2} = \left[ {\begin{array}{*{20}{c}}x&y\end{array}} \right]\left[ {\begin{array}{*{20}{c}}a&h\\h&b\end{array}} \right]\left[ {\begin{array}{*{20}{c}}x\\y\end{array}} \right]$$But the matrix could be diagonalized as $$\left[ {\begin{array}{*{20}{c}}a&h\\h&b\end{array}} \right] = M\left[ {\begin{array}{*{20}{c}}{\frac{{a + b - \sqrt {{{(a - b)}^2} + 4{h^2}} }}{2}}&0\\0&{\frac{{a + b + \sqrt {{{(a - b)}^2} + 4{h^2}} }}{2}}\end{array}} \right]{M^T}$$ (however irrelevant to your question, note that $$M = \left[ {\begin{array}{*{20}{c}}{\frac{{a - b - \sqrt {{{(a - b)}^2} + 4{h^2}} }}{{2h\sqrt {1 + {{\left( {\frac{{a - b - \sqrt {{{(a - b)}^2} + 4{h^2}} }}{{2h}}} \right)}^2}} }}}&{\frac{{a - b + \sqrt {{{(a - b)}^2} + 4{h^2}} }}{{2h\sqrt {1 + {{\left( {\frac{{a - b + \sqrt {{{(a - b)}^2} + 4{h^2}} }}{{2h}}} \right)}^2}} }}}\\{\frac{1}{{\sqrt {1 + {{\left( {\frac{{a - b - \sqrt {{{(a - b)}^2} + 4{h^2}} }}{{2h}}} \right)}^2}} }}}&{\frac{1}{{\sqrt {1 + {{\left( {\frac{{a - b + \sqrt {{{(a - b)}^2} + 4{h^2}} }}{{2h}}} \right)}^2}} }}}\end{array}} \right]$$)Now, by introducing the new variables $${M^T}\left[ {\begin{array}{*{20}{c}}x\\y\end{array}} \right] = \left[ {\begin{array}{*{20}{c}}{x'}\\{y'}\end{array}} \right]$$we have $$P(x',y') = \left[ {\begin{array}{*{20}{c}}{x'}&y\end{array}'} \right]\left[ {\begin{array}{*{20}{c}}{\frac{{a + b - \sqrt {{{(a - b)}^2} + 4{h^2}} }}{2}}&0\\0&{\frac{{a + b + \sqrt {{{(a - b)}^2} + 4{h^2}} }}{2}}\end{array}} \right]\left[ {\begin{array}{*{20}{c}}{x'}\\{y'}\end{array}} \right]$$ Clearly for the polynomial to be of determined sign, both values on the diagonal should have the same sign. Now you should be able to figure out the conditions. Good luck ;)