I am attempting to find the rotated equation for a conic section.
However, the formula used to find the angle rotated, $cot(2\theta)=\frac{A-C}{B}$ (see more info here) for the polynomial $Ax^2 + Bxy + Cy^2 + Dx + Ey + F = 0$ does not work for certain conic sections such as:
$x^2-10xy+y^2+1=0$
When plugging in the $\frac{A-C}{B}$ terms we get $\frac{1-1}{-10}=0$ which means that $\theta=0$, but this doesn't make any sense.
This conic section is clearly rotated, as it has a $Bxy$ term, and its graph looks rotated.
Are there any other methods that I can use to determine the angle of rotation, or am I doing something incorrectly here?
You may be getting the cotangent function confounded with the tangent function.
It is not true that $\cot(2\theta)=0$ implies $\theta=0.$
In fact, $\cot(0)$ is not defined, so the formula should never yield $\theta=0.$
But $\cot\left(\frac\pi2\right)=0$ and $\cot\left(\frac32\pi\right)=0,$ so finding that $\cot(2\theta)=0$ is an indication that $\theta = \frac\pi4$ or $\theta=\frac34\pi,$ either of which is a possible interpretation for $x^2-10xy+y^2+1=0$ (depending on whether you make the $x$ axis or the $y$ axis the major axis before rotating the hyperbola).