show that the straight lines $(a^2-3b^2)x^2+8abxy+(b^2-3a^2)y^2=0$ form with the lie ax+by+c=o an equilateral triangle whose area is $\frac{c^2}{\sqrt{3}(a^2+b^2)}$
is there any other way to solve without intersection points. Also the Angle between the first two lines is 60 degrees.any another way to solve this.
As I was saying, if we have three constants $F,G,H$ and $$ F x^2 + G x y + H y^2 = 0, $$ we divide through by $x^2$ to arrive at $$ F + G \frac{y}{x} + H \frac{y^2}{x^2} = 0. $$ Or, as I said, using the traditional $m = \frac{y}{x},$ we get $$ F + G m + H m^2 = 0,$$ or in order $$ H m^2 + G m + F = 0.$$
We suspect we are describing two lines passing through the origin. If so, the slope(s) $m$ satisfy $$ m = \frac{-G \pm \sqrt {G^2 - 4 HF}}{2H}. $$ So, if $G^2 - 4 F H > 0,$ we do have two different slopes and two lines. For your original problem $$ F = a^2 - 3 b^2, G = 8 a b, H = b^2 - 3 a^2 $$