let $$A = \left(\begin{array}{cccc} 1&2&3\\2&3&4\\3&4&5 \end{array}\right)$$
I need to find an invertible matrix $P$ such that $P^tAP$ is a diagonal matrix and it's main diagonal may have only the terms from the set $\{ 1,-1,0 \}$
I'd be glad if you could explain to me how to solve this. I haven't found the right theorem/algorithm.
Thanks.
This trick is due to Hermite. It is especially useful when you have a symmetric matrix of integers. First, we write a certain function in three variables, $$ f(x,y,z) = x^2 + 3 y^2 + 5 z^2 + 8 yz+ 6 zx +4xy, $$ because this is exactly the result of calculating $v^t A v,$ with $$ v = \left( \begin{array}{c} x \\ y \\ z \end{array} \right) $$
In order to clear the terms with $x,$ we write $$ (x + 2 y + 3 z)^2 = x^2 + 4 y^2 + 9 z^2 + 12 yz+ 6 zx +4xy. $$ So far, $$ f(x,y,z) - (x + 2 y + 3 z)^2 = -y^2 - 4 z^2 - 4 y z. $$ Next we clear all $y,$ $$ (y + 2 z)^2 = y^2 + 4 z^2 + 4 y z, $$ and $$ f(x,y,z) - (x + 2 y + 3 z)^2 + (y + 2 z)^2 = 0, $$ $$ \color{red}{ f(x,y,z) = (x + 2 y + 3 z)^2 - (y + 2 z)^2 }. $$
The matrix multiplication that this shows is $$ \left( \begin{array}{ccc} 1 & 0 & 0 \\ 2 & 1 & 0 \\ 3 & 2 & 0 \end{array} \right) \left( \begin{array}{ccc} 1 & 0 & 0 \\ 0 & -1 & 0 \\ 0 & 0 & 0 \end{array} \right) \left( \begin{array}{ccc} 1 & 2 & 3 \\ 0 & 1 & 2 \\ 0 & 0 & 0 \end{array} \right) = \left( \begin{array}{ccc} 1 & 2 & 3 \\ 2 & 3 & 4 \\ 3 & 4 & 5 \end{array} \right) $$ That is actually the right way to do it.
However, I see that someone asked for invertible $P,$ despite non-full rank. Also can be done, and easily: $$ \left( \begin{array}{ccc} 1 & 0 & 0 \\ 2 & 1 & 0 \\ 3 & 2 & 1 \end{array} \right) \left( \begin{array}{ccc} 1 & 0 & 0 \\ 0 & -1 & 0 \\ 0 & 0 & 0 \end{array} \right) \left( \begin{array}{ccc} 1 & 2 & 3 \\ 0 & 1 & 2 \\ 0 & 0 & 1 \end{array} \right) = \left( \begin{array}{ccc} 1 & 2 & 3 \\ 2 & 3 & 4 \\ 3 & 4 & 5 \end{array} \right) $$ The effect of this is to add $0$ to the function in the shape of $0z^2.$
ADDED: looking at the question again, we do need the extra $1$ in the lower right, because the matrix $P$ requested is actually the inverse of the on I display above. Life goes on.