Let $A=\begin{pmatrix} 0 & -2 & 1 \\ -2 & 1 & 2 \\ 1 & 2 & 0 \end{pmatrix} \in M_3(\mathbb{R})$.
I want to find an invertible matrix $C$ such that $C^TAC=\begin{pmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & -1 \end{pmatrix}$, since there are three eigenvalues, two of them are positive, one is negative.
I want to use completing the square with the bilinear form $s(v,v)=\langle v,Av \rangle$:
$s(v,v)= \langle v,Av \rangle=(v_1,v_2,v_3)\begin{pmatrix} 0 & -2 & 1 \\ -2 & 1 & 2 \\ 1 & 2 & 0 \end{pmatrix}\begin{pmatrix} v_1\\v_2\\v_3 \end{pmatrix}=-4v_1v_2+2v_1v_3+v_2^2+4v_2v_3$
Now I tried to use the formula $ax^2+bx=a(x+\frac{b}{2a})^2-\frac{b^2}{2a}$, but there is no $v_1^2$.
So I started with $v_2^2$:
$s(v,v)=(v_2+2v_3)^2-4v_1v_2+2v_1v_3-4v_3^2$
Here I don't see how to use completing the square again with $-4v_1v_2+2v_1v_3$, since $v_1^2$ is not available. Is there a method how to continue?
Given the expression below, multiply on the right and left by symmetric $$ B = \left( \begin{array}{rrr} 1 & 0 & 0 \\ 0 & \frac{1}{2} & 0 \\ 0 & 0 & \frac{ 2 }{ 3 } \\ \end{array} \right) $$ to get your desired matrix as $B^T P^T HPB = (PB)^T H (PB).$ Actually, this puts the diagonal entries in order $1,-1,1.$ A final step is to multiply on left and right by a permutation matrix that interchanges rows (or columns) 2,3.
$$ P^T H P = D $$ $$\left( \begin{array}{rrr} 0 & 1 & 0 \\ 1 & 2 & 0 \\ \frac{ 5 }{ 4 } & \frac{ 1 }{ 2 } & 1 \\ \end{array} \right) \left( \begin{array}{rrr} 0 & - 2 & 1 \\ - 2 & 1 & 2 \\ 1 & 2 & 0 \\ \end{array} \right) \left( \begin{array}{rrr} 0 & 1 & \frac{ 5 }{ 4 } \\ 1 & 2 & \frac{ 1 }{ 2 } \\ 0 & 0 & 1 \\ \end{array} \right) = \left( \begin{array}{rrr} 1 & 0 & 0 \\ 0 & - 4 & 0 \\ 0 & 0 & \frac{ 9 }{ 4 } \\ \end{array} \right) $$
Algorithm discussed at http://math.stackexchange.com/questions/1388421/reference-for-linear-algebra-books-that-teach-reverse-hermite-method-for-symmetr
https://en.wikipedia.org/wiki/Sylvester%27s_law_of_inertia
$$ H = \left( \begin{array}{rrr} 0 & - 2 & 1 \\ - 2 & 1 & 2 \\ 1 & 2 & 0 \\ \end{array} \right) $$ $$ D_0 = H $$ $$ E_j^T D_{j-1} E_j = D_j $$ $$ P_{j-1} E_j = P_j $$ $$ E_j^{-1} Q_{j-1} = Q_j $$ $$ P_j Q_j = Q_j P_j = I $$ $$ P_j^T H P_j = D_j $$ $$ Q_j^T D_j Q_j = H $$
$$ H = \left( \begin{array}{rrr} 0 & - 2 & 1 \\ - 2 & 1 & 2 \\ 1 & 2 & 0 \\ \end{array} \right) $$
==============================================
$$ E_{1} = \left( \begin{array}{rrr} 0 & 1 & 0 \\ 1 & 0 & 0 \\ 0 & 0 & 1 \\ \end{array} \right) $$ $$ P_{1} = \left( \begin{array}{rrr} 0 & 1 & 0 \\ 1 & 0 & 0 \\ 0 & 0 & 1 \\ \end{array} \right) , \; \; \; Q_{1} = \left( \begin{array}{rrr} 0 & 1 & 0 \\ 1 & 0 & 0 \\ 0 & 0 & 1 \\ \end{array} \right) , \; \; \; D_{1} = \left( \begin{array}{rrr} 1 & - 2 & 2 \\ - 2 & 0 & 1 \\ 2 & 1 & 0 \\ \end{array} \right) $$
==============================================
$$ E_{2} = \left( \begin{array}{rrr} 1 & 2 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \\ \end{array} \right) $$ $$ P_{2} = \left( \begin{array}{rrr} 0 & 1 & 0 \\ 1 & 2 & 0 \\ 0 & 0 & 1 \\ \end{array} \right) , \; \; \; Q_{2} = \left( \begin{array}{rrr} - 2 & 1 & 0 \\ 1 & 0 & 0 \\ 0 & 0 & 1 \\ \end{array} \right) , \; \; \; D_{2} = \left( \begin{array}{rrr} 1 & 0 & 2 \\ 0 & - 4 & 5 \\ 2 & 5 & 0 \\ \end{array} \right) $$
==============================================
$$ E_{3} = \left( \begin{array}{rrr} 1 & 0 & - 2 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \\ \end{array} \right) $$ $$ P_{3} = \left( \begin{array}{rrr} 0 & 1 & 0 \\ 1 & 2 & - 2 \\ 0 & 0 & 1 \\ \end{array} \right) , \; \; \; Q_{3} = \left( \begin{array}{rrr} - 2 & 1 & 2 \\ 1 & 0 & 0 \\ 0 & 0 & 1 \\ \end{array} \right) , \; \; \; D_{3} = \left( \begin{array}{rrr} 1 & 0 & 0 \\ 0 & - 4 & 5 \\ 0 & 5 & - 4 \\ \end{array} \right) $$
==============================================
$$ E_{4} = \left( \begin{array}{rrr} 1 & 0 & 0 \\ 0 & 1 & \frac{ 5 }{ 4 } \\ 0 & 0 & 1 \\ \end{array} \right) $$ $$ P_{4} = \left( \begin{array}{rrr} 0 & 1 & \frac{ 5 }{ 4 } \\ 1 & 2 & \frac{ 1 }{ 2 } \\ 0 & 0 & 1 \\ \end{array} \right) , \; \; \; Q_{4} = \left( \begin{array}{rrr} - 2 & 1 & 2 \\ 1 & 0 & - \frac{ 5 }{ 4 } \\ 0 & 0 & 1 \\ \end{array} \right) , \; \; \; D_{4} = \left( \begin{array}{rrr} 1 & 0 & 0 \\ 0 & - 4 & 0 \\ 0 & 0 & \frac{ 9 }{ 4 } \\ \end{array} \right) $$
==============================================
$$ P^T H P = D $$ $$\left( \begin{array}{rrr} 0 & 1 & 0 \\ 1 & 2 & 0 \\ \frac{ 5 }{ 4 } & \frac{ 1 }{ 2 } & 1 \\ \end{array} \right) \left( \begin{array}{rrr} 0 & - 2 & 1 \\ - 2 & 1 & 2 \\ 1 & 2 & 0 \\ \end{array} \right) \left( \begin{array}{rrr} 0 & 1 & \frac{ 5 }{ 4 } \\ 1 & 2 & \frac{ 1 }{ 2 } \\ 0 & 0 & 1 \\ \end{array} \right) = \left( \begin{array}{rrr} 1 & 0 & 0 \\ 0 & - 4 & 0 \\ 0 & 0 & \frac{ 9 }{ 4 } \\ \end{array} \right) $$ $$ Q^T D Q = H $$ $$\left( \begin{array}{rrr} - 2 & 1 & 0 \\ 1 & 0 & 0 \\ 2 & - \frac{ 5 }{ 4 } & 1 \\ \end{array} \right) \left( \begin{array}{rrr} 1 & 0 & 0 \\ 0 & - 4 & 0 \\ 0 & 0 & \frac{ 9 }{ 4 } \\ \end{array} \right) \left( \begin{array}{rrr} - 2 & 1 & 2 \\ 1 & 0 & - \frac{ 5 }{ 4 } \\ 0 & 0 & 1 \\ \end{array} \right) = \left( \begin{array}{rrr} 0 & - 2 & 1 \\ - 2 & 1 & 2 \\ 1 & 2 & 0 \\ \end{array} \right) $$