My professor was trying to explain finding minimum and maximum using second derivative test. He started by considering
$$ \begin{matrix} W(x,y)=ax^2+bxy+cy^2 \\ =a(x^2+\frac{b}{a}xy)+cy^2\\ =a(x^2+2\cdot x \cdot\frac{b}{2a}y+(\frac{by}{2a})^2)+cy^2-\frac{b^2 y^2}{4a} \\ =a(x+\frac{by}{2a})^2-(c-\frac{b^2}{4a})y^2\\ =\frac{1}{4a}[(4a^2(x+\frac{by}{2a})^2)+(4ac-b^2)y^2)] \ \ \ \text{pulling out a common factor of } \ \frac{1}{4a} \end{matrix} $$
So this whole expression yields a form of sum and difference of two squares depending on the value of $(4ac-b^2)$
When he came to the case where $(4ac-b^2)>0$, he said that the expression was of the form identical to the sum of two squares, but then he inferred whether this represented a max or min depended on the sign of a and if $a>0$ it meant minimum and if $a<0$ it meant maximum. I want to understand how he reached this conclusion
Not sure how detailed your discussion was, but I think your professor was trying to explain something along the lines of this:
If you have some two variable function $f(x,y)$, you can create a taylor approximation to near some fixed point $(x_0,y_0)$ like so:
$$f(x_0 + x,y_0+y) \\ =f(x_0,y_0) + \partial_x f(x_0,y_0)x + \partial_y f(x_0,y_0)x + \frac{1}{2}\partial_{xx} f(x_0,y_0)x^2 + \partial_{xx} f(x_0,y_0)xy + \frac{1}{2}\partial_{yy} f(x_0,y_0)y^2 + O(x^3)$$ Where $O(x^3)$ is the error between the approximation of $f$ and the 2nd order polynomial we constructed. Now at an extrema, $\partial_x f = \partial_y f = 0$, so if we assume $(x_0,y_0)$ is an extrema, we can get rid of a couple of terms in the above expression. We're left with the equation $$f(x_0+x,y_0+y) - f(x_0,y_0) = \frac{1}{2}\partial_{xx} f(x_0,y_0)x^2 + \partial_{xx} f(x_0,y_0)xy + \frac{1}{2}\partial_{yy} f(x_0,y_0)y^2 + O(x^3)$$ Now if we substitute $a = \frac{1}{2}\partial_{xx} f(x_0,y_0), b = \partial_{xy} f(x_0,y_0), c = \frac{1}{2}\partial_{yy} f(x_0,y_0)$. The last equation becomes: $$f(x_0+x,y_0+y) - f(x_0,y_0) = ax^2 + bxy + cy^2 + O(x^3) \\ = \frac{1}{4a}[(4a^2(x+\frac{by}{2a})^2)+(4ac-b^2)y^2)] + O(x^3)$$ Notice that in this last expression, we see that $a>0$ and $4ac-b^2>0$ forces the RHS to be positive, so for a sufficiently close enough $f(x_0+x,y_0+y)$, the LHS comes out to always positive. So all nearby points are larger than $f(x_0,y_0)$ therefore we have that $f(x_0,y_0)$ must be a minimum. If instead $a<0$, similar reason shows we have a maximum. If $4ac - b^2 < 0$, then regardless of the sign of $a$, we can choose $x$ and $y$ to make the RHS positive or negative, so that means we have neither a maximum or a minimum. We must instead have a saddle point. Does that make sense?