The question is
Show that the two lines given by $$(A^2 - 3B^2)x^2 + 8ABxy +(B^2 - 3A^2)y^2=0$$ and the line given by $$Ax+By+C=0$$ determine an equilateral triangle of area $$\frac{C^2}{\sqrt{3}\;(A^2+B^2)}$$
I tried factorizing the pair of straight lines into two straight lines. But that seemed to be not really simple. And I doubt I was proceeding rightly. It would be great if anyone helps.
On
i will need the following preliminary results: $\begin{align} & \text{ the angle between the lines } ax^2 + bxy + cy^2 = 0 \text{ is $60^\circ$ iff } b^2 = 3a^2+3c^2 + 10ac.\tag 1\\ & \text{ the bisectors of } ax^2 + bxy + cy^2 = 0 \text{ are } bx^2 - 2(c-a)xy -by^2 = 0\tag 2\end{align}$
proof: suppose the angle between the lines $$ 0 = (a_1x - b_1y)( a_2x - b_2y) = a_1a_2x^2 -(b_2a_1+b_1a_2)xy + b_1b_2y^2 = ax^2 + bxy + cy^2 $$ is $60^\circ.$ let $t_1, t_2$ be defined by $\tan t_2 = a_2/b_2, \tan t_1 = a_1/b_1.$ then
$$3 = \tan^2 (t_2 - t_1) = \left(\dfrac{\tan t_2 - \tan t_1}{1 + \tan t_2 \tan t_1}\right)^2 = \left(\dfrac{b_2a_1 - b_1a_2}{a_1a_2 + b_1b_2}\right)^2 = \frac{(b_2a_1+b_1a_2)^2 - 4a_1a_2b_1b_2)}{(a_1a_2+b_1b_2)^2} = \dfrac{b^2-4ac}{(a+c)^2}.$$ that proves the first claim.
proof of the second claim. pick a point $(x,y).$ then equation the distances to the two lines we get $$\frac{(a_1-b_1)^2}{a_1^2 + b_1^2} = \frac{(a_1-b_1)^2}{a_1^2 + b_1^2} $$ cleaning this up proves the second claim.
we also want to establish that $$\text{ angle between the lines } (A^2 - 3B^2) x^2 - 8ABxy + (B^2 - 3A^2) y^2 = 0 \text{ is 60^\circ and are bisected by the lines } (Ax + By)(Bx-Ay) = 0 \tag 3$$
next, given the bisectors $(Ax + By)(Bx-Ay)$ we will use $(1)$ and $2$ to determine the lines $ax^2 + bxy + cy^2 = 0$ so that angles between them is $60^\circ$
by (2) $bx^2 - 2(c-a)xy -by^2 = 0$ is equivalent to $ABx^2 +(B^2-A^2)xy -ABy^2 = 0$we will set $$b = 2kAB, a-c = k(B^2 - A^2) \tag 4$$ now im]ose the constraint (1) that angle between them is $60^\circ.$ that gives $$4k^2 A^2B^2 = 3a^2 + 3c^2 + 10ac=3(a-c)^2 + 16ac=3k^2(B^2-A^2)^2+16ac $$ we can rewrite this as $$16ac = k^2\left(4A^2B^2-3(B^2-A^2)^2\right)\tag 5$$ we will determine $-a$ and $c$ as the roots of the quadratic equation $$ 0 = (x+a)(x-c) =x^2 + x(a-c) - ac = x^2+k(B^2-A^2)x - \frac1{16}k^2(4A^2B^2-3(B^2-A^2)^2)$$ we will set $k = 2$ for convenience. the discriminant $D$ is $$ D = 4(B^2-A^2)^2 + 4A^2B^2-3(B^2-A^2)^2 = (B^2+A^2)^2.$$ therefore $$ c = \frac{(3A^2-B^2)}2, a = \frac{(3B^2-A^2)}2 \text{ or } c = \frac{(A^2-3B^2)}2, a = \frac{(B^2-3A^2)}2$$
therefore we have now established that $ \dfrac{(3B^2-A^2)}2 x^2 + 4ABxy + \dfrac{(3A^2-B^2)}2 y^2 = 0 $ this is equivalent to the claim (3).
we now come to final part of finding the area of the equilateral triangle bounded by $ (A^2 - 3B^2) x^2 - 8ABxy + (B^2 - 3A^2) y^2 = 0 $ and $Ax + By = C = 0.$ let $d$ be the distance of the line $Ax + By + C = 0$ from the origin. then $d^2 = \dfrac{C^2}{A^2 + B^2}$ and the area of the equilateral triangle is $$\text{ area of the triangle = }\dfrac{d^2}{\sqrt 3} = \dfrac{C^2}{\sqrt 3(A^2 + B^2)}.$$
note that it does not matter which bisector we used because the area of the equilateral triangle and the triangle with angles $120^\circ, 30\circ, 30^\circ$ have the same area.
On
The crossed lines defined by $$( A^2 - 3 B^2 ) x^2 + 8 A B x y + ( B^2 - 3 A^2 ) y^2 = 0 \qquad(\star)$$ pass through the origin, so one of our vertices is $O(0,0)$.
Convenient manipulation allows us to re-write $(\star)$ thusly: $$x^2 + y^2 = \frac{4( Ax + B y )^2}{3(A^2+B^2)} \qquad(\star\star)$$ If $P(x,y)$ satisfies $(\star)$ (that is, $(\star\star)$) and also $A x + B y + C = 0$, then we have $$x^2 + y^2 = \frac{4C^2}{3(A^2+B^2)} \quad\text{so that}\quad |\overline{OP}| = \frac{2|C|}{\sqrt{3}\;\sqrt{A^2+B^2}}$$
But $d := |C|/\sqrt{A^2+B^2}$ is precisely the distance from $O$ to the line defined by $Ax+By+C=0$. Consequently, if $P_1$ and $P_2$ are the other two vertices of our triangle, then $$|\overline{OP_1}| = |\overline{OP_2}| = \frac{2}{\sqrt{3}}d$$ That is, $\triangle OP_1P_2$ is isosceles. Moreover, the ratio of its base altitude ($d$) to its leg makes it equilateral, and we can write $$|\triangle OP_1P_2| = \frac{1}{2} \;d\;|\overline{OP_1}| = \frac{1}{2} d^2 \frac{2}{\sqrt{3}} = \frac{C^2}{\sqrt{3}\;(A^2+B^2)}$$ as desired. $\square$
On
$$( A^2 - 3 B^2 ) x^2 + 8 A B x y + ( B^2 - 3 A^2 ) y^2 = 0 $$
are a pair of straight lines through the the origin making angle $ \pi/3 $ or $2 \pi/3 $ with each other at the origin, just like two sides along diameter of circum-circle of a regular hexagon.
They are factorisable into two straight lines as it is easy to verify that for two cases in particular :
$$ A = 1, B=0 \rightarrow x^2 - 3 y^2 = 0 ; A = 0, B=1 \rightarrow y^2 - 3 x^2 = 0 $$
and, in general,
$$ (A^2-3 B^2) x^2 - 8 A B x y + ( B^2 - 3 A^2 ) y^2 = 0 $$ has slopes
$$ y = m_1 x , y = m_2 x ; \,( m_1, m_2) = \frac {4\, A B \pm \sqrt{3}(A^2 +B^2 )}{ (A^2 -B^2 )} $$
A simple way: let us apply the similarity transform $u=Ax+By,v=Bx-Ay$.
The line equation becomes $$Ax+By+C=u+C=0$$ and the line pair equation
$$(A^2-3B^2)x^2+8ABxy+(B^2-3A^2)y^2=u^2-3v^2=(u-\sqrt3v)(u+\sqrt3v)=0.$$
The pairwise intersections are $(0,0),(-C,\dfrac C{\sqrt3}),(-C,-\dfrac C{\sqrt3})$, thus the three squared side lengths are $\dfrac43C^2$ and the triangle is equilateral.
Correcting by the similarity ratio, $$Area=\frac{\sqrt3}4\left(\frac43C^2\right)\left(\frac1{\sqrt{A^2+B^2}}\right)^2=\frac{C^2}{\sqrt3(A^2+B^2)}.$$