If I have the coefficients of the following equation:
$$AX^2 + BXY + CY^2 + DX + EY + F = 0$$
And I know it's a hyperbola, how can I get the equations for the asymptotes with respect to the coefficients A, B, C, D, E, and F?
i.e. similar to this question, except that question deals with properties of ellipses from the general equation instead of hyperbolas.
On
The asymptotes of a hyperbola are the tangents at its intersections with the line at infinity. The matrix of this general equation is $$Q = \begin{bmatrix}A&\frac B2&\frac D2\\\frac B2&C&\frac E2\\\frac D2&\frac E2&F\end{bmatrix}$$ and the line at infinity is $\mathscr l = [0,0,1]^T$. Let $\mathscr l_\times$ be the “cross product matrix” of $\mathscr l$. We can compute the hyperbola-line intersection by finding a value of $\alpha$ that makes $$\mathscr l_\times^TQ\mathscr l_\times+\alpha\mathscr l_\times = \begin{bmatrix}C&-\alpha-\frac B2&0\\\alpha-\frac B2&A&0\\0&0&0\end{bmatrix}$$ a rank-one matrix. This occurs when the principal $2\times2$ minor vanishes, which leads to a quadratic equation in $\alpha$ with solutions $\pm\frac12\sqrt{B^2-4AC}$. Taking the positive root gives the matrix $$\begin{bmatrix}C&-\frac12\left(B+\sqrt{B^2-4AC}\right)&0 \\ -\frac12\left(B^2-\sqrt{B^2-4AC}\right)&A&0\\0&0&0\end{bmatrix}.$$ The two intersection points are the row and column corresponding to any nonzero element of this matrix. For example, if $C\ne0$, then the two points are $\mathbf p_1 = \left[C,-\frac12\left(B+\sqrt{B^2-4AC}\right),0\right]^T$ and $\mathbf p_2 = \left[C,-\frac12\left(B-\sqrt{B^2-4AC}\right),0\right]^T$, the projective equivalents of the asymptotes’ direction vectors, with corresponding tangents $\mathscr m_i = Q\mathbf p_i$. The implicit Cartesian equations of the lines that these vectors represent are $[x,y,1]\mathscr m_i=0$, or $[x,y,1]Q\mathbf p_i=0$.
On
The equation of a hyperbola whose asymptotes have equations $ax+by+c=0$ and $dx+ey+f=0$ can be written as: $$ (ax+by+c)(dx+ey+f)=g. $$ You only need then to expand the above equation and compare the coefficient of the resulting polynomial with $A\dots F$ to obtain $a\dots g$.
In practice, to avoid redundancies, it's convenient to divide the equation of the hyperbola by $A$ (if $A\ne0$) and thus set $a=d=1$, to get a system of five equations in five unknowns.
On
In matrix form the equation is $$x^TAx+2b^Tx+c=0$$ where $A$ is $2\times2$ symmetric.
It can be centered with
$$(x+A^{-1}b)^TA(x+A^{-1}b)-b^TA^{-T}A^Tb+c=y^TAy+c'=0.$$
Then, by diagonalizing $A$,
$$y^TP\Lambda P^{-1}y+c'=z^T\Lambda z+c'=0.$$
For an hyperbola, the Eigenvalues have opposite signs and the reduced equation is
$$\lambda_uu^2-\lambda_vv^2=(\sqrt{\lambda_u}u+\sqrt{\lambda_v}v)(\sqrt{\lambda_u}u-\sqrt{\lambda_v}v)=-c'.$$
The two factors are the asymptotes, and in the original coordinates
$$x=P^{-1}z-A^{-1}b.$$
First find the centre of the conic.
This is the point $(u,v),$ such that the equation of the conic can be rewritten as $$A(X-u)^2+B(X-u)(Y-v)+C(Y-v)^2+F'=0.$$ For the conic to be a hyperbola, the quadratic part has to factor into distinct linear factors over $\Bbb R$: $$AX^2+BXY+CY^2=(R_1X+S_1Y)(R_2X+S_2Y).$$
Then the asymptotes are $$R_i (X-u)+S_i(Y-v)=0$$ ($i=1$, $2$).