Let the following be a ternary quadratic form:
$$A = \begin{pmatrix}0 & a & b \\ a & 0 & c \\ b & c & 0 \end{pmatrix}$$
with $a,b,c\in\mathbb{Q}$. If at least one term in the diagonal was nonzero, one could transform it into a canonical form by "completing squares".
How would one proceed in this case?
In essence, I need to find two matrices $R$ and $D$ where $D$ is diagonal so that $R^TDR = A$. How would one find such matrices?
Also, could one ensure that the entries on $R$ and $D$ are rational?
There is no agreed upon canonical form for indefinite ternary integer/rational quadratic forms. In the order you requested,
$$ \left( \begin{array}{rrr} \frac{1}{2} & -1 & 0 \\ \frac{1}{2} & 1 & 0 \\ \frac{b+c}{2a} & \frac{b-c}{a} & 1 \end{array} \right) \left( \begin{array}{rrr} 2 a & 0 & 0 \\ 0 & -\frac{a}{2} & 0 \\ 0 & 0 & - \frac{2bc}{a} \end{array} \right) \left( \begin{array}{rrr} \frac{1}{2} & \frac{1}{2} & \frac{b+c}{2a} \\ -1 & 1 & \frac{b-c}{a} \\ 0 & 0 & 1 \end{array} \right) = \left( \begin{array}{rrr} 0 & a & b \\ a & 0 & c \\ b & c & 0 \end{array} \right) $$
See reference for linear algebra books that teach reverse Hermite method for symmetric matrices where you are using the inverse of the matrix found in their algorithm.