Equation of surface $$5x_1^2+2x_2^2+2x_3^2-{4x_1 x_2}-{4x_1 x_3}-{8x_2 x_3}=7.$$
I ended up with $$(2x_1 - x_2 - x_3)^2 + (x_1 + x_2 + x_3)^2 - 2x_1 x_2 - 2x_1 x_3 - 8x_2 x_3 = 7$$ but don't know how to proceed. Would be highly apppreciated if you could complete the square for this equation so that I can see if the eigenvalues are positive or not. Thank you so much! :)
I do this with symmetric matrices. The first thing I did was to switch $x_1$ and $x_2.$ Let me take a minute to show just that much in matrices...
Note that I change one $\pm$ sign to keep $\det P = 1$
$$ \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc $$
$$ P^T H P = D $$
$$\left( \begin{array}{rrr} 0 & 1 & 0 \\ - 1 & 0 & 0 \\ 0 & 0 & 1 \\ \end{array} \right) \left( \begin{array}{rrr} 5 & - 2 & - 2 \\ - 2 & 2 & - 4 \\ - 2 & - 4 & 2 \\ \end{array} \right) \left( \begin{array}{rrr} 0 & - 1 & 0 \\ 1 & 0 & 0 \\ 0 & 0 & 1 \\ \end{array} \right) = \left( \begin{array}{rrr} 2 & 2 & - 4 \\ 2 & 5 & 2 \\ - 4 & 2 & 2 \\ \end{array} \right) $$ $$ $$
$$ Q^T D Q = H $$
$$\left( \begin{array}{rrr} 0 & - 1 & 0 \\ 1 & 0 & 0 \\ 0 & 0 & 1 \\ \end{array} \right) \left( \begin{array}{rrr} 2 & 2 & - 4 \\ 2 & 5 & 2 \\ - 4 & 2 & 2 \\ \end{array} \right) \left( \begin{array}{rrr} 0 & 1 & 0 \\ - 1 & 0 & 0 \\ 0 & 0 & 1 \\ \end{array} \right) = \left( \begin{array}{rrr} 5 & - 2 & - 2 \\ - 2 & 2 & - 4 \\ - 2 & - 4 & 2 \\ \end{array} \right) $$
From this point it became easy
$$ \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc $$
$$ \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc $$
$$ P^T H P = D $$
$$\left( \begin{array}{rrr} 0 & 1 & 0 \\ - 1 & - 1 & 0 \\ 2 & 4 & 1 \\ \end{array} \right) \left( \begin{array}{rrr} 5 & - 2 & - 2 \\ - 2 & 2 & - 4 \\ - 2 & - 4 & 2 \\ \end{array} \right) \left( \begin{array}{rrr} 0 & - 1 & 2 \\ 1 & - 1 & 4 \\ 0 & 0 & 1 \\ \end{array} \right) = \left( \begin{array}{rrr} 2 & 0 & 0 \\ 0 & 3 & 0 \\ 0 & 0 & - 18 \\ \end{array} \right) $$ $$ $$
$$ Q^T D Q = H $$
$$\left( \begin{array}{rrr} - 1 & - 1 & 0 \\ 1 & 0 & 0 \\ - 2 & 2 & 1 \\ \end{array} \right) \left( \begin{array}{rrr} 2 & 0 & 0 \\ 0 & 3 & 0 \\ 0 & 0 & - 18 \\ \end{array} \right) \left( \begin{array}{rrr} - 1 & 1 & - 2 \\ - 1 & 0 & 2 \\ 0 & 0 & 1 \\ \end{array} \right) = \left( \begin{array}{rrr} 5 & - 2 & - 2 \\ - 2 & 2 & - 4 \\ - 2 & - 4 & 2 \\ \end{array} \right) $$
$$ \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc $$
The outcome, as formulas, is that your form (in variables $x,y,z$) comes out $$ \color{blue}{2(-x+y-2z)^2 + 3 (-x+2z)^2 -18 z^2} $$
If you do the thing purely algorithmically you must expect fractions...
Algorithm discussed at reference for linear algebra books that teach reverse Hermite method for symmetric matrices
https://en.wikipedia.org/wiki/Sylvester%27s_law_of_inertia
$$ H = \left( \begin{array}{rrr} 5 & - 2 & - 2 \\ - 2 & 2 & - 4 \\ - 2 & - 4 & 2 \\ \end{array} \right) $$ $$ D_0 = H $$ $$ E_j^T D_{j-1} E_j = D_j $$ $$ P_{j-1} E_j = P_j $$ $$ E_j^{-1} Q_{j-1} = Q_j $$ $$ P_j Q_j = Q_j P_j = I $$ $$ P_j^T H P_j = D_j $$ $$ Q_j^T D_j Q_j = H $$
$$ H = \left( \begin{array}{rrr} 5 & - 2 & - 2 \\ - 2 & 2 & - 4 \\ - 2 & - 4 & 2 \\ \end{array} \right) $$
==============================================
$$ E_{1} = \left( \begin{array}{rrr} 1 & \frac{ 2 }{ 5 } & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \\ \end{array} \right) $$ $$ P_{1} = \left( \begin{array}{rrr} 1 & \frac{ 2 }{ 5 } & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \\ \end{array} \right) , \; \; \; Q_{1} = \left( \begin{array}{rrr} 1 & - \frac{ 2 }{ 5 } & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \\ \end{array} \right) , \; \; \; D_{1} = \left( \begin{array}{rrr} 5 & 0 & - 2 \\ 0 & \frac{ 6 }{ 5 } & - \frac{ 24 }{ 5 } \\ - 2 & - \frac{ 24 }{ 5 } & 2 \\ \end{array} \right) $$
==============================================
$$ E_{2} = \left( \begin{array}{rrr} 1 & 0 & \frac{ 2 }{ 5 } \\ 0 & 1 & 0 \\ 0 & 0 & 1 \\ \end{array} \right) $$ $$ P_{2} = \left( \begin{array}{rrr} 1 & \frac{ 2 }{ 5 } & \frac{ 2 }{ 5 } \\ 0 & 1 & 0 \\ 0 & 0 & 1 \\ \end{array} \right) , \; \; \; Q_{2} = \left( \begin{array}{rrr} 1 & - \frac{ 2 }{ 5 } & - \frac{ 2 }{ 5 } \\ 0 & 1 & 0 \\ 0 & 0 & 1 \\ \end{array} \right) , \; \; \; D_{2} = \left( \begin{array}{rrr} 5 & 0 & 0 \\ 0 & \frac{ 6 }{ 5 } & - \frac{ 24 }{ 5 } \\ 0 & - \frac{ 24 }{ 5 } & \frac{ 6 }{ 5 } \\ \end{array} \right) $$
==============================================
$$ E_{3} = \left( \begin{array}{rrr} 1 & 0 & 0 \\ 0 & 1 & 4 \\ 0 & 0 & 1 \\ \end{array} \right) $$ $$ P_{3} = \left( \begin{array}{rrr} 1 & \frac{ 2 }{ 5 } & 2 \\ 0 & 1 & 4 \\ 0 & 0 & 1 \\ \end{array} \right) , \; \; \; Q_{3} = \left( \begin{array}{rrr} 1 & - \frac{ 2 }{ 5 } & - \frac{ 2 }{ 5 } \\ 0 & 1 & - 4 \\ 0 & 0 & 1 \\ \end{array} \right) , \; \; \; D_{3} = \left( \begin{array}{rrr} 5 & 0 & 0 \\ 0 & \frac{ 6 }{ 5 } & 0 \\ 0 & 0 & - 18 \\ \end{array} \right) $$
==============================================
$$ \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc $$
$$ P^T H P = D $$
$$\left( \begin{array}{rrr} 1 & 0 & 0 \\ \frac{ 2 }{ 5 } & 1 & 0 \\ 2 & 4 & 1 \\ \end{array} \right) \left( \begin{array}{rrr} 5 & - 2 & - 2 \\ - 2 & 2 & - 4 \\ - 2 & - 4 & 2 \\ \end{array} \right) \left( \begin{array}{rrr} 1 & \frac{ 2 }{ 5 } & 2 \\ 0 & 1 & 4 \\ 0 & 0 & 1 \\ \end{array} \right) = \left( \begin{array}{rrr} 5 & 0 & 0 \\ 0 & \frac{ 6 }{ 5 } & 0 \\ 0 & 0 & - 18 \\ \end{array} \right) $$ $$ $$
$$ Q^T D Q = H $$
$$\left( \begin{array}{rrr} 1 & 0 & 0 \\ - \frac{ 2 }{ 5 } & 1 & 0 \\ - \frac{ 2 }{ 5 } & - 4 & 1 \\ \end{array} \right) \left( \begin{array}{rrr} 5 & 0 & 0 \\ 0 & \frac{ 6 }{ 5 } & 0 \\ 0 & 0 & - 18 \\ \end{array} \right) \left( \begin{array}{rrr} 1 & - \frac{ 2 }{ 5 } & - \frac{ 2 }{ 5 } \\ 0 & 1 & - 4 \\ 0 & 0 & 1 \\ \end{array} \right) = \left( \begin{array}{rrr} 5 & - 2 & - 2 \\ - 2 & 2 & - 4 \\ - 2 & - 4 & 2 \\ \end{array} \right) $$
$$ \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc $$