The sides of a triangle have the combined equation $x^2-3y^2-2xy+8y-4=0.$ The third side, which is variable always passes through the point $(-5,-1)$ . If the range of values of slope of the third line so that origin is an interior point of the triangle, lies in the interval $(a,b)$. Then $\bigg(a+\frac{1}{b^2}\bigg)$ is ?
What I tried:
$3y^2-2(x-4)y+4-x^2=0$
$\displaystyle y =\frac{+2(x-4)\pm \sqrt{4(x-2)^2-12(4-x^2)}}{6}$
$\displaystyle y =\frac{(x-4)\pm \sqrt{4x^2-4x+16}}{3}$
and equation of third side is $y+1=m(x+5)$
How do I solve it? Help me, please!
HINT
The combined equation of the two sides $x^2-3y^2-2xy+8y-4=0$ can be written as $(x-3y+2)(x+y-2)=0,$ hence the corresponding equations are $$\begin{aligned}x-3y+2&=0\\x+y-2&=0\end{aligned}$$ The situation is as shows the figure.
Note that the point $M(-5,-1)$ lies on one of the given sides, hence it is a vertex of the triangle.
The auxiliary line through $M$ and origin has the slope which is the highest limit value.
The lowest limit value (negative) of the slope through $M$ is obtained, when this side is "parallel" to the line with equation $x+y-2=0.$