Consider $Ax^{2}+By^{2}+Cxy+Dx+Ey+F=0 \space \space (*)$
In the case when $C=1$, find, in terms of the coefficients of (*), the value of the constant $k$ for which the transformation with the matrix
\begin{pmatrix} k & 1 \\ -1 & k \end{pmatrix}
reduces (*) to a form with zero $xy$-term.
I let the image point be \begin{pmatrix} x' \\ y' \end{pmatrix}
and so the pre-image is
\begin{pmatrix} \frac{kx'-y'}{k^{2}+1} \\ \frac{x'+ky'}{k^{2}+1} \end{pmatrix}
Substituting this into (*) and set the coefficient of $xy$-term to zero. I managed to obtain $k=A-B\pm\sqrt{(A-B)^{2}+1}$.
Is there a way to solve this problem using eigenvalues and eigenvectors?
We find a matrix $P$ which diagonalizes the symmetric matrix $A=\begin{pmatrix}A&\frac C2\\\frac C2&B\end{pmatrix}$. Then, the transformation $\begin{bmatrix} x\\y\end{bmatrix}=P\begin{bmatrix} x'\\y'\end{bmatrix}$ eliminates the $xy$-term, because $P^T$ is a multiple of $P^{-1}$ for real symmetric matrices. Recall that the associated quadratic form is $\begin{bmatrix} x\\y\end{bmatrix}^TA\begin{bmatrix} x\\y\end{bmatrix}$.
Example: Let us consider the ellipse $2x^2+2xy+y^2=1\tag1$ Then $A=\begin{pmatrix}2&1\\1&1\end{pmatrix}$ with eigen values $\lambda_1=\phi+1$ and $\lambda_2=2-\phi$ and corresponding eigenvectors $v_1=\begin{bmatrix}\phi\\1\end{bmatrix}$ and $v_2=\begin{bmatrix}1\\-\phi\end{bmatrix}$. Thus $P=\begin{pmatrix}\phi&1\\1&-\phi\end{pmatrix}$. Now, $x=\phi x'+ y'$ and $y=x'-\phi y'$ transform the given ellipse to $(\phi+1)(x')^2+(2-\phi)(y')^2=\frac1{\phi+2}\tag2$ with no cross-term.