For a 2x2 symmetric rational matrix $$ M = \begin{bmatrix} a & b \\ b & c \end{bmatrix} $$ how can we characterize the invertible matrices of rational numbers $$ P = \begin{bmatrix} r & s \\ t & u \end{bmatrix} $$ $$ M' = P^T M P $$ which will result in $M'$ diagonal?
(EDIT: There is already in error in my first step here, see comment by Ruy.) There is clearly a freedom in the overall scaling, so we can write $P = q S$ for non-zero $q \in \mathbb{Q}$ and $S \in \text{SL}(2,\mathbb{Q})$.
I can see that once the matrix is diagonal, we can scale the individual basis vectors and keep it diagonal, so we also have the freedom in $SL(2,Q)$: $$ \begin{bmatrix} k & 0 \\ 0 & 1/k \end{bmatrix} $$ where $k$ is rational.
With some playing I found that this matrix would make $P^T M P$ diagonal and is in $SL(2,Q)$: $$ \begin{bmatrix} 1 & -b/a \\ 0 & 1 \end{bmatrix} $$
So at least a subset of $P$ which result in $M'=P^T MP$ diagonal can be characterized by two rational numbers $q$ and $k$: $$ P = q \begin{bmatrix} k & 0 \\ 0 & 1/k \end{bmatrix} \begin{bmatrix} 1 & -b/a \\ 0 & 1 \end{bmatrix} $$
Is this the full characterization?
I'm a bit confused, because another way to go about this would be to find the eigenvalues and eigenvectors, and since $M$ is symmetric this would diagonalize the matrix. But I don't quite see how this relates to the solution I have above. So maybe there is another freedom?
Looking around, this note on SL(2,R) seems to suggest there may be a "rotation" like element that also diagonalizes. So maybe there is another freedom that I am missing?
An explicit example to help me think this out: $$M = \begin{bmatrix} 2 & 1/2 \\ 1/2 & 3 \end{bmatrix}$$ with eigen-system $$ \begin{aligned} \lambda_1 &= \frac{1}{2}(5+\sqrt{2}), \quad v_1 = \begin{bmatrix}-1+\sqrt{2} \\ 1 \end{bmatrix} \\ \lambda_2 &= \frac{1}{2}(5-\sqrt{2}), \quad v_2 = \begin{bmatrix}-1-\sqrt{2} \\ 1 \end{bmatrix} \\ \end{aligned} $$ Then we have $$ \begin{aligned} P &= \begin{bmatrix} -1 + \sqrt{2} & -1-\sqrt{2} \\ 1 & 1 \end{bmatrix} \\ P^T M P &= \begin{bmatrix} 8-3\sqrt{2} & 0 \\ 0 & 8 + 3\sqrt{2} \end{bmatrix} \\ \end{aligned} $$ Because of the $\sqrt{2}$ these are not rational. But since this seems like the canonical way to diagonalize, it seems like it must be related to my other method above via rescaling or something. Unfortunately I am having trouble seeing how, and so furthering my confusion here.