I tried using the formula $\cot(2\theta)=\dfrac{A-C}{B}$, and got it down to $A-C=2$.
I would need a second equation to find the values of those coefficients. So I tried this instead:
To remove $xy$ terms from the equation, we use the formulas$$ \begin{cases} x=x'\cos(\theta)-y'\sin(\theta)\\ y=x'\sin(\theta)+y'\cos(\theta) \end{cases}.$$ After simplifying and substituting into the equation, I got this: $$A=C+6\sqrt{3}-\frac{13}{2},\ A=C-\frac{7}{2}, C=\frac{A}{4}-\frac{3}{8}+\frac{3C}{4}-\frac{3\sqrt{3}}{2}+2\sqrt{3}.$$
But when I tried solving these equations, I get "no solutions".
What is the valid method of finding the values of $A$ and $C$?