Find the condition that the expressions $ax^2+2hxy+by^2$,$a'x^2+2h'xy+b'y^2$ may be respectively divisible by factors of the form $y-mx$, $my+x$ .
What I've tried:
Let $p(x,y) = ax^2 + 2hxy + by^2$ and $q(x,y) = a'x^2 + 2h'xy +b'y^2$ and $y-mx$, $my+x$ be factors of $p(x,y)$ and $q(x,y)$ respectively.
Now putting $y = mx$ in $p(x,y)$ , I got : $x^2(a + 2hm + bm^2x^2) = 0$ or, $bm^2 + 2hm + a = 0$ ...(i)
And putting $x = -my$ in $q(x,y)$ , I got : $y^2(a'm^2 - 2h'm + b') = 0$ or, $a'm^2 - 2h'm + b' = 0$ ...(ii)
From (i) and (ii), $aa' + 2hma' + 2h'mb - bb' = 0$
But the required condition is $(aa' - bb')^2 + 4(ha'+h'b)(hb' + h'a) = 0$
From \begin{eqnarray} bm^2+2hm+a& =& 0 \\a'm^2-2h'm+b' &=& 0, \end{eqnarray} we need to eliminate $m$ to get a single equation in the coefficients. We could do this directly by solving one equation for $m$ using the quadratic formula, then plugging that value into the other. However, the square roots will make the algebra in that second step messy. You can try it if you like--it'll get the same solution--but there's a better way.
First, note that if $b m^2 + 2hm + a = 0$ and $a'm^2 - 2h'm + b' = 0$, then it is also true that $p ( m^2 + 2hm + a) + q (a'm^2 - 2h'm + b') = 0$ for any $p,q$. With the proper choice of $p$ and $q$, we can simplify the dependence on $m$ by getting rid of either the linear term or the quadratic term. To get rid of the quadratic term, we use $p = a'$ and $q = -b$. To get rid of the linear term, we use $p = h'$ and $q=h$. This gives \begin{eqnarray} 2(a'h+bh')m + aa'-bb' &=& 0\\ (a'h+ bh')m^2 + ah'+b'h &=& 0. \end{eqnarray} Multiplying the second equation by $4(a'h + bh')$ and rearranging the first gives the more suggestive form \begin{eqnarray} 2(a'h+bh')m &=& bb'-aa'\\ \left[2(a'h+ bh')m\right]^2 &+& 4(a'h + bh')(ah'+b'h)= 0. \end{eqnarray} From which we clearly have $$ \left(bb'-aa'\right)^2 + 4(a'h + bh')(ah'+b'h)= 0, $$ which is the same as the given result because $(bb'-aa')^2 = (aa'-bb')^2$.