In my lecture notes, I have the following solution.
Find canonical form and classify the conic section $x^2+8y^2-4xy-8x+6y-5=0$.
$x^2+8y^2-4xy-8x+6y-5=\frac{4}{109}(x-2y-4)^2+\frac{16}{109}(y-\frac54)^2-1=0$. Letting $X:=x-2y-4, Y:=y-\frac54$ we get the canonical form and it is an equation of ellipse.
How can we choose $X:=x-2y-4, Y:=y-\frac54$ arbitrarily and decide the canonical form? I did not understand this way. What is the mathematical reason?
Thanks in advance.

Something is wrong in your version. The $ Q^T D Q = H $ version below says your polynomial is $$ (x-2y - 4)^2 + 4 (y - \frac{5}{4})^2 - \frac{109}{4} $$ I get it, you multiplied through by $\frac{4}{109}$
Meanwhile, there is no particular canonical form. One may complete some squares and get a diagonalized matrix as shown. Meanwhile, there is an algorithm that works, see the referenced question.
$$ \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc $$
$$ P^T H P = D $$
$$\left( \begin{array}{rrr} 1 & 0 & 0 \\ 2 & 1 & 0 \\ \frac{ 13 }{ 2 } & \frac{ 5 }{ 4 } & 1 \\ \end{array} \right) \left( \begin{array}{rrr} 1 & - 2 & - 4 \\ - 2 & 8 & 3 \\ - 4 & 3 & - 5 \\ \end{array} \right) \left( \begin{array}{rrr} 1 & 2 & \frac{ 13 }{ 2 } \\ 0 & 1 & \frac{ 5 }{ 4 } \\ 0 & 0 & 1 \\ \end{array} \right) = \left( \begin{array}{rrr} 1 & 0 & 0 \\ 0 & 4 & 0 \\ 0 & 0 & - \frac{ 109 }{ 4 } \\ \end{array} \right) $$ $$ $$
$$ Q^T D Q = H $$
$$\left( \begin{array}{rrr} 1 & 0 & 0 \\ - 2 & 1 & 0 \\ - 4 & - \frac{ 5 }{ 4 } & 1 \\ \end{array} \right) \left( \begin{array}{rrr} 1 & 0 & 0 \\ 0 & 4 & 0 \\ 0 & 0 & - \frac{ 109 }{ 4 } \\ \end{array} \right) \left( \begin{array}{rrr} 1 & - 2 & - 4 \\ 0 & 1 & - \frac{ 5 }{ 4 } \\ 0 & 0 & 1 \\ \end{array} \right) = \left( \begin{array}{rrr} 1 & - 2 & - 4 \\ - 2 & 8 & 3 \\ - 4 & 3 & - 5 \\ \end{array} \right) $$
$$ \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc $$
Algorithm discussed at reference for linear algebra books that teach reverse Hermite method for symmetric matrices
https://en.wikipedia.org/wiki/Sylvester%27s_law_of_inertia
$$ H = \left( \begin{array}{rrr} 1 & - 2 & - 4 \\ - 2 & 8 & 3 \\ - 4 & 3 & - 5 \\ \end{array} \right) $$ $$ D_0 = H $$ $$ E_j^T D_{j-1} E_j = D_j $$ $$ P_{j-1} E_j = P_j $$ $$ E_j^{-1} Q_{j-1} = Q_j $$ $$ P_j Q_j = Q_j P_j = I $$ $$ P_j^T H P_j = D_j $$ $$ Q_j^T D_j Q_j = H $$
$$ H = \left( \begin{array}{rrr} 1 & - 2 & - 4 \\ - 2 & 8 & 3 \\ - 4 & 3 & - 5 \\ \end{array} \right) $$
==============================================
$$ E_{1} = \left( \begin{array}{rrr} 1 & 2 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \\ \end{array} \right) $$ $$ P_{1} = \left( \begin{array}{rrr} 1 & 2 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \\ \end{array} \right) , \; \; \; Q_{1} = \left( \begin{array}{rrr} 1 & - 2 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \\ \end{array} \right) , \; \; \; D_{1} = \left( \begin{array}{rrr} 1 & 0 & - 4 \\ 0 & 4 & - 5 \\ - 4 & - 5 & - 5 \\ \end{array} \right) $$
==============================================
$$ E_{2} = \left( \begin{array}{rrr} 1 & 0 & 4 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \\ \end{array} \right) $$ $$ P_{2} = \left( \begin{array}{rrr} 1 & 2 & 4 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \\ \end{array} \right) , \; \; \; Q_{2} = \left( \begin{array}{rrr} 1 & - 2 & - 4 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \\ \end{array} \right) , \; \; \; D_{2} = \left( \begin{array}{rrr} 1 & 0 & 0 \\ 0 & 4 & - 5 \\ 0 & - 5 & - 21 \\ \end{array} \right) $$
==============================================
$$ E_{3} = \left( \begin{array}{rrr} 1 & 0 & 0 \\ 0 & 1 & \frac{ 5 }{ 4 } \\ 0 & 0 & 1 \\ \end{array} \right) $$ $$ P_{3} = \left( \begin{array}{rrr} 1 & 2 & \frac{ 13 }{ 2 } \\ 0 & 1 & \frac{ 5 }{ 4 } \\ 0 & 0 & 1 \\ \end{array} \right) , \; \; \; Q_{3} = \left( \begin{array}{rrr} 1 & - 2 & - 4 \\ 0 & 1 & - \frac{ 5 }{ 4 } \\ 0 & 0 & 1 \\ \end{array} \right) , \; \; \; D_{3} = \left( \begin{array}{rrr} 1 & 0 & 0 \\ 0 & 4 & 0 \\ 0 & 0 & - \frac{ 109 }{ 4 } \\ \end{array} \right) $$
==============================================
$$ \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc $$
$$ P^T H P = D $$
$$\left( \begin{array}{rrr} 1 & 0 & 0 \\ 2 & 1 & 0 \\ \frac{ 13 }{ 2 } & \frac{ 5 }{ 4 } & 1 \\ \end{array} \right) \left( \begin{array}{rrr} 1 & - 2 & - 4 \\ - 2 & 8 & 3 \\ - 4 & 3 & - 5 \\ \end{array} \right) \left( \begin{array}{rrr} 1 & 2 & \frac{ 13 }{ 2 } \\ 0 & 1 & \frac{ 5 }{ 4 } \\ 0 & 0 & 1 \\ \end{array} \right) = \left( \begin{array}{rrr} 1 & 0 & 0 \\ 0 & 4 & 0 \\ 0 & 0 & - \frac{ 109 }{ 4 } \\ \end{array} \right) $$ $$ $$
$$ Q^T D Q = H $$
$$\left( \begin{array}{rrr} 1 & 0 & 0 \\ - 2 & 1 & 0 \\ - 4 & - \frac{ 5 }{ 4 } & 1 \\ \end{array} \right) \left( \begin{array}{rrr} 1 & 0 & 0 \\ 0 & 4 & 0 \\ 0 & 0 & - \frac{ 109 }{ 4 } \\ \end{array} \right) \left( \begin{array}{rrr} 1 & - 2 & - 4 \\ 0 & 1 & - \frac{ 5 }{ 4 } \\ 0 & 0 & 1 \\ \end{array} \right) = \left( \begin{array}{rrr} 1 & - 2 & - 4 \\ - 2 & 8 & 3 \\ - 4 & 3 & - 5 \\ \end{array} \right) $$
$$ \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc $$