We want to find when a quadratic form depending on a parameter $\alpha$ on $\mathbb{R}^3$ is an inner product. It is quite obvious and quick to deal with the problem by using Sylvester, BUT we want to do it with our bare hands, without resorting to this criterion or to any related result.
I am horrible at completing squares, and I am managing to not get the desired result even if I know from Sylvester that the correct answer is that $\alpha$ needs to be greater than golden ratio $\frac{1+\sqrt{5}}{2}$.
The form has matrix $\left(\begin{array}{ccc} 1 & -1 & 0 \\ -1 & \alpha & 1\\ 0 & 1 & \alpha\end{array}\right)$, so what we actually want to find out is the real values of $\alpha$ such that the polynomial
$$x_1^2 + \alpha (x_2^2 + x_3^2) + 2x_2(x_3-x_1) $$ is strictly positive for all $(x_1,x_2,x_3)$ other than $(0,0,0)$.
There is an algorithm for taking a symmetric matrix $H$ and finding a nonsingular matrix $P$ such that $P^T H P$ is diagonal. A different way of completing the square(s), no guesswork needed. Then Sylvester's Law of Inertia says that the eigenvalues of that $D,$ those being the diagonal entries, have the same counts of positive, negative, zero eigenvalues.
See reference for linear algebra books that teach reverse Hermite method for symmetric matrices
When $a \neq 1 \; : \; \; $
$$ P = \left( \begin{array}{rrr} 1&1& \frac{-1}{a-1} \\ 0&1& \frac{-1}{a-1} \\ 0&0&1 \\ \end{array} \right) $$
$$ P^T H P = \left( \begin{array}{rrr} 1&0& 0 \\ 0&a-1& 0 \\ 0&0& \frac{a^2 - a - 1}{a-1} \\ \end{array} \right) $$
When are the three diagonal elements positive?
$$ \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc $$
Here is the algorithm applied when fixing $a=8$
$$ H = \left( \begin{array}{rrr} 1 & - 1 & 0 \\ - 1 & 8 & 1 \\ 0 & 1 & 8 \\ \end{array} \right) $$ $$ D_0 = H $$ $$ E_j^T D_{j-1} E_j = D_j $$ $$ P_{j-1} E_j = P_j $$ $$ E_j^{-1} Q_{j-1} = Q_j $$ $$ P_j Q_j = Q_j P_j = I $$ $$ P_j^T H P_j = D_j $$ $$ Q_j^T D_j Q_j = H $$
$$ H = \left( \begin{array}{rrr} 1 & - 1 & 0 \\ - 1 & 8 & 1 \\ 0 & 1 & 8 \\ \end{array} \right) $$
==============================================
$$ E_{1} = \left( \begin{array}{rrr} 1 & 1 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \\ \end{array} \right) $$ $$ P_{1} = \left( \begin{array}{rrr} 1 & 1 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \\ \end{array} \right) , \; \; \; Q_{1} = \left( \begin{array}{rrr} 1 & - 1 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \\ \end{array} \right) , \; \; \; D_{1} = \left( \begin{array}{rrr} 1 & 0 & 0 \\ 0 & 7 & 1 \\ 0 & 1 & 8 \\ \end{array} \right) $$
==============================================
$$ E_{2} = \left( \begin{array}{rrr} 1 & 0 & 0 \\ 0 & 1 & - \frac{ 1 }{ 7 } \\ 0 & 0 & 1 \\ \end{array} \right) $$ $$ P_{2} = \left( \begin{array}{rrr} 1 & 1 & - \frac{ 1 }{ 7 } \\ 0 & 1 & - \frac{ 1 }{ 7 } \\ 0 & 0 & 1 \\ \end{array} \right) , \; \; \; Q_{2} = \left( \begin{array}{rrr} 1 & - 1 & 0 \\ 0 & 1 & \frac{ 1 }{ 7 } \\ 0 & 0 & 1 \\ \end{array} \right) , \; \; \; D_{2} = \left( \begin{array}{rrr} 1 & 0 & 0 \\ 0 & 7 & 0 \\ 0 & 0 & \frac{ 55 }{ 7 } \\ \end{array} \right) $$
==============================================
$$ \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc $$
$$ P^T H P = D $$
$$\left( \begin{array}{rrr} 1 & 0 & 0 \\ 1 & 1 & 0 \\ - \frac{ 1 }{ 7 } & - \frac{ 1 }{ 7 } & 1 \\ \end{array} \right) \left( \begin{array}{rrr} 1 & - 1 & 0 \\ - 1 & 8 & 1 \\ 0 & 1 & 8 \\ \end{array} \right) \left( \begin{array}{rrr} 1 & 1 & - \frac{ 1 }{ 7 } \\ 0 & 1 & - \frac{ 1 }{ 7 } \\ 0 & 0 & 1 \\ \end{array} \right) = \left( \begin{array}{rrr} 1 & 0 & 0 \\ 0 & 7 & 0 \\ 0 & 0 & \frac{ 55 }{ 7 } \\ \end{array} \right) $$ $$ $$
$$ Q^T D Q = H $$
$$\left( \begin{array}{rrr} 1 & 0 & 0 \\ - 1 & 1 & 0 \\ 0 & \frac{ 1 }{ 7 } & 1 \\ \end{array} \right) \left( \begin{array}{rrr} 1 & 0 & 0 \\ 0 & 7 & 0 \\ 0 & 0 & \frac{ 55 }{ 7 } \\ \end{array} \right) \left( \begin{array}{rrr} 1 & - 1 & 0 \\ 0 & 1 & \frac{ 1 }{ 7 } \\ 0 & 0 & 1 \\ \end{array} \right) = \left( \begin{array}{rrr} 1 & - 1 & 0 \\ - 1 & 8 & 1 \\ 0 & 1 & 8 \\ \end{array} \right) $$
$$ \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc $$
When $a=2$
$$ H = \left( \begin{array}{rrr} 1 & - 1 & 0 \\ - 1 & 2 & 1 \\ 0 & 1 & 2 \\ \end{array} \right) $$ $$ D_0 = H $$ $$ E_j^T D_{j-1} E_j = D_j $$ $$ P_{j-1} E_j = P_j $$ $$ E_j^{-1} Q_{j-1} = Q_j $$ $$ P_j Q_j = Q_j P_j = I $$ $$ P_j^T H P_j = D_j $$ $$ Q_j^T D_j Q_j = H $$
$$ H = \left( \begin{array}{rrr} 1 & - 1 & 0 \\ - 1 & 2 & 1 \\ 0 & 1 & 2 \\ \end{array} \right) $$
==============================================
$$ E_{1} = \left( \begin{array}{rrr} 1 & 1 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \\ \end{array} \right) $$ $$ P_{1} = \left( \begin{array}{rrr} 1 & 1 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \\ \end{array} \right) , \; \; \; Q_{1} = \left( \begin{array}{rrr} 1 & - 1 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \\ \end{array} \right) , \; \; \; D_{1} = \left( \begin{array}{rrr} 1 & 0 & 0 \\ 0 & 1 & 1 \\ 0 & 1 & 2 \\ \end{array} \right) $$
==============================================
$$ E_{2} = \left( \begin{array}{rrr} 1 & 0 & 0 \\ 0 & 1 & - 1 \\ 0 & 0 & 1 \\ \end{array} \right) $$ $$ P_{2} = \left( \begin{array}{rrr} 1 & 1 & - 1 \\ 0 & 1 & - 1 \\ 0 & 0 & 1 \\ \end{array} \right) , \; \; \; Q_{2} = \left( \begin{array}{rrr} 1 & - 1 & 0 \\ 0 & 1 & 1 \\ 0 & 0 & 1 \\ \end{array} \right) , \; \; \; D_{2} = \left( \begin{array}{rrr} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \\ \end{array} \right) $$
==============================================
$$ \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc $$
$$ P^T H P = D $$
$$\left( \begin{array}{rrr} 1 & 0 & 0 \\ 1 & 1 & 0 \\ - 1 & - 1 & 1 \\ \end{array} \right) \left( \begin{array}{rrr} 1 & - 1 & 0 \\ - 1 & 2 & 1 \\ 0 & 1 & 2 \\ \end{array} \right) \left( \begin{array}{rrr} 1 & 1 & - 1 \\ 0 & 1 & - 1 \\ 0 & 0 & 1 \\ \end{array} \right) = \left( \begin{array}{rrr} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \\ \end{array} \right) $$ $$ $$
$$ Q^T D Q = H $$
$$\left( \begin{array}{rrr} 1 & 0 & 0 \\ - 1 & 1 & 0 \\ 0 & 1 & 1 \\ \end{array} \right) \left( \begin{array}{rrr} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \\ \end{array} \right) \left( \begin{array}{rrr} 1 & - 1 & 0 \\ 0 & 1 & 1 \\ 0 & 0 & 1 \\ \end{array} \right) = \left( \begin{array}{rrr} 1 & - 1 & 0 \\ - 1 & 2 & 1 \\ 0 & 1 & 2 \\ \end{array} \right) $$
$$ \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc $$
When $a=1$
$$ H = \left( \begin{array}{rrr} 1 & - 1 & 0 \\ - 1 & 1 & 1 \\ 0 & 1 & 1 \\ \end{array} \right) $$ $$ D_0 = H $$ $$ E_j^T D_{j-1} E_j = D_j $$ $$ P_{j-1} E_j = P_j $$ $$ E_j^{-1} Q_{j-1} = Q_j $$ $$ P_j Q_j = Q_j P_j = I $$ $$ P_j^T H P_j = D_j $$ $$ Q_j^T D_j Q_j = H $$
$$ H = \left( \begin{array}{rrr} 1 & - 1 & 0 \\ - 1 & 1 & 1 \\ 0 & 1 & 1 \\ \end{array} \right) $$
==============================================
$$ E_{1} = \left( \begin{array}{rrr} 1 & 1 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \\ \end{array} \right) $$ $$ P_{1} = \left( \begin{array}{rrr} 1 & 1 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \\ \end{array} \right) , \; \; \; Q_{1} = \left( \begin{array}{rrr} 1 & - 1 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \\ \end{array} \right) , \; \; \; D_{1} = \left( \begin{array}{rrr} 1 & 0 & 0 \\ 0 & 0 & 1 \\ 0 & 1 & 1 \\ \end{array} \right) $$
==============================================
$$ E_{2} = \left( \begin{array}{rrr} 1 & 0 & 0 \\ 0 & 0 & 1 \\ 0 & 1 & 0 \\ \end{array} \right) $$ $$ P_{2} = \left( \begin{array}{rrr} 1 & 0 & 1 \\ 0 & 0 & 1 \\ 0 & 1 & 0 \\ \end{array} \right) , \; \; \; Q_{2} = \left( \begin{array}{rrr} 1 & - 1 & 0 \\ 0 & 0 & 1 \\ 0 & 1 & 0 \\ \end{array} \right) , \; \; \; D_{2} = \left( \begin{array}{rrr} 1 & 0 & 0 \\ 0 & 1 & 1 \\ 0 & 1 & 0 \\ \end{array} \right) $$
==============================================
$$ E_{3} = \left( \begin{array}{rrr} 1 & 0 & 0 \\ 0 & 1 & - 1 \\ 0 & 0 & 1 \\ \end{array} \right) $$ $$ P_{3} = \left( \begin{array}{rrr} 1 & 0 & 1 \\ 0 & 0 & 1 \\ 0 & 1 & - 1 \\ \end{array} \right) , \; \; \; Q_{3} = \left( \begin{array}{rrr} 1 & - 1 & 0 \\ 0 & 1 & 1 \\ 0 & 1 & 0 \\ \end{array} \right) , \; \; \; D_{3} = \left( \begin{array}{rrr} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & - 1 \\ \end{array} \right) $$
==============================================
$$ \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc $$
$$ P^T H P = D $$
$$\left( \begin{array}{rrr} 1 & 0 & 0 \\ 0 & 0 & 1 \\ 1 & 1 & - 1 \\ \end{array} \right) \left( \begin{array}{rrr} 1 & - 1 & 0 \\ - 1 & 1 & 1 \\ 0 & 1 & 1 \\ \end{array} \right) \left( \begin{array}{rrr} 1 & 0 & 1 \\ 0 & 0 & 1 \\ 0 & 1 & - 1 \\ \end{array} \right) = \left( \begin{array}{rrr} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & - 1 \\ \end{array} \right) $$ $$ $$
$$ Q^T D Q = H $$
$$\left( \begin{array}{rrr} 1 & 0 & 0 \\ - 1 & 1 & 1 \\ 0 & 1 & 0 \\ \end{array} \right) \left( \begin{array}{rrr} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & - 1 \\ \end{array} \right) \left( \begin{array}{rrr} 1 & - 1 & 0 \\ 0 & 1 & 1 \\ 0 & 1 & 0 \\ \end{array} \right) = \left( \begin{array}{rrr} 1 & - 1 & 0 \\ - 1 & 1 & 1 \\ 0 & 1 & 1 \\ \end{array} \right) $$
$$ \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc $$