Here $B^T$ denotes the transpose of $B$.
$A$ and $B$ are invertible $3\times 3$ matrices with integer entries.
$A$ is symmetric positive definite with at most two zero entries.
We want the determinants of both $A$ and $B$ be as small as possible (in terms of absolute value)
For a fixed matrix $A$, we want to find all matrices $B$ with small determinants, preferably less than 10, if they exist.
On
two completely different questions.........
$$ \left( \begin{array}{rrr} 1 & 0 & 0 \\ -1 & 1 & 0 \\ 0 & -1 & 1 \end{array} \right) \left( \begin{array}{rrr} 1 & 1 & 1 \\ 1 & 2 & 2 \\ 1 & 2 & 3 \end{array} \right) \left( \begin{array}{rrr} 1 & -1 & 0 \\ 0 & 1 & -1 \\ 0 & 0 & 1 \end{array} \right) = \left( \begin{array}{rrr} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{array} \right) $$
You should be aware that, for any symmetric matrix $A$ of integers, there is a rational matrix $B$ of determinant $\pm 1$ such that $B^T A B$ is diagonal. See reference for linear algebra books that teach reverse Hermite method for symmetric matrices
If, as appears, you are writing a computer program to solve the second question: it is more difficult than you think. I suggest that you solve it completely for the 2 by 2 case; I guess this means a working program. You will be able to find numerous references on equivalence of binary quadratic forms over the rationals and over the integers. For the three by three case, start with my introduction.
If A is a symmetric positive definite matrix then $B^TAB$ is a diagonal matrix iff B is an orthogonal matrix i.e. $B^T=B^{-1}$. If $\exists$ such a matrix $B$ then $B^TAB=diag(\lambda_1,\lambda_2,...,\lambda_n)$ where $\lambda_i$ are the eigenvalues of $A$. And the columns of $B$ are the corresponding eigenvectors of $A$. So to find the $B$ s.t. $det(B)<10$ you will need to find the eigenvectors of A.