Q: Let us be given the parabola $(Ax + By)^2 + 2gx + 2fy + c = 0$ s.t. $A^2 + B^2 \neq 0$. Then what is the directrix and focus for the parabola when $Af - Bg = 0$?
Background. Consider that equation of the parabola, we want to write it as the distance of $(x,y)$ from some line equals distance from some point (the line and point being the directrix and focus respectively). [Let K be a parameter which can take any value.]
\begin{align*} (Ax + By)^2 + 2gx + 2fy + c &= 0 \\ \Leftrightarrow B^2 x^2 + A^2 y^2 - 2AB xy + 2BKx - 2AKy + K^2 + \Big(\frac{(g+BK)^2}{A^2 + B^2} + \frac{(f-AK)^2}{A^2 + B^2} - c - K^2 \Big) = \\ (A^2+B^2)\Big( (x + \frac{g + BK}{A^2 + B^2})^2 + (y + \frac{f - AK}{A^2 + B^2})^2 \Big) \\ \Leftrightarrow (Bx - Ay + K)^2 + \epsilon(K) = (A^2+B^2)\Big( (x + \frac{g + BK}{A^2 + B^2})^2 + (y + \frac{f - AK}{A^2 + B^2})^2 \Big) \\ \end{align*} where $\epsilon(K) := \Big(\frac{(g+BK)^2}{A^2 + B^2} + \frac{(f-AK)^2}{A^2 + B^2} - c - K^2 \Big)$. Now above equation holds for any value of $K$, and we chose that value of K s.t. $\epsilon(K) = 0$ (as notice once we have $\epsilon(K) = 0$, we have found the directrix and focus), and we get the following result:
If $Af - Bg \neq 0$, then $K = \frac{(A^2 +B^2)c - (g^2 + f^2)}{2(Af - Bg)} \Rightarrow \epsilon(K) = 0$.
But this now leaves the important question, what if $Af - Bg = 0$, how do we find the directrix and focus?
Let $r = [ x, y]^T $ then your parabola is given by
$ r^T Q r + r^T L + c = 0 $
where
$ Q = \begin{bmatrix} A^2 && A B \\ A B && B^2 \end{bmatrix} $
$L = \begin{bmatrix} 2 g \\ 2 f \end{bmatrix} $
Diagonalize $Q$. To do that, calculate the angle $\theta$ as follows
$ \tan (2 \theta) = \dfrac{ 2 AB }{A^2 - B^2} $
So that
$R = \begin{bmatrix} \cos \theta && - \sin \theta \\ \sin \theta && \cos \theta \end{bmatrix} $
Now,
$ \cos^2(2 \theta) = \dfrac{1}{\tan^2(2 \theta) + 1} = \dfrac{ (A^2 - B^2)^2 }{(A^2 + B^2)^2 } $
So, $\cos(2 \theta) = \dfrac{ |A^2 - B^2| }{A^2 + B^2} $
We can simply take $ \cos(2 \theta) = \dfrac{ A^2 - B^2 }{A^2 + B^2} $ and maintain $\tan(2 \theta) $ at the above value,then
$\sin(2 \theta) = \tan(2 \theta) \cos(2 \theta) = \dfrac{2 A B}{A^2 + B^2} $
If $Q = R D R^T$. The diagonal entries of $D$ are
$D_{11} = \dfrac{1}{2} (A^2 + B^2) + \dfrac{1}{2} (A^2 - B^2) \cos(2 \theta) + AB \sin(2 \theta) $
And this is equal to
$ D_{11} = \dfrac{1}{2}(A^2 + B^2) + \dfrac{1}{2} \dfrac{(A^2 - B^2)^2}{A^2 + B^2} + \dfrac{ 2 A^2 B^2}{A^2 + B^2}$
And this simplifies to
$D_{11} = \dfrac{1}{2} \dfrac{A^4 + 4 A^2 B^2 + B^4}{A^2 + B^2} $
while
$D_{22} = 0 $
Back to the expression for $\cos(2\theta) = \dfrac{A^2 - B^2}{A^2 + B^2} $
It follows that
$ \cos(\theta) = \sqrt{ \dfrac{1 + \cos(2 \theta)}{2} } = \dfrac{A}{\sqrt{A^2 + B^2}} $
$ \sin(\theta) = \sqrt{ \dfrac{ 1 - \cos(2 \theta)}{2} } = \dfrac{B}{\sqrt{A^2+B^2} } $
From this,
$ \tan(2 \theta) = \dfrac{2 \tan(\theta)}{1 - \tan^2(\theta)} = \dfrac{2 \sin(\theta)\cos(\theta) }{\cos^2(\theta) - \sin^2(\theta)} = \dfrac{2 AB}{A^2 - B^2} $
Back to the original equation
$ r^T Q r + r^T L + c = 0 $
with $Q = R D R^T $ this becomes
$ r^T R D R^T r + r^T L + c = 0 $
Let $w = R^T r$ , then
$ w^T D w + w^T R^T L + c = 0 $
Now
$ R^T L = \dfrac{1}{\sqrt{A^2 + B^2}} \begin{bmatrix} A && B \\ -B && A \end{bmatrix} \begin{bmatrix} 2 g\\ 2 f \end{bmatrix} = \dfrac{ 2}{\sqrt{A^2 + B^2}} \begin{bmatrix} A g + B f \\ 0 \end{bmatrix}$
So in this case, we get a degenerate equation
$ D_{11} w_1^2 + \dfrac{2}{\sqrt{A^2 + B^2}} (A g + B f) w_1 + c = 0 $
which can have 0, 1, or 2 solutions, while $w_2$ is arbitrary, and this corresponds to no solution, one line, or two parallel lines.