We are asked to find the number of $3$x$3$ matrices $A$ with all integer entries satisfying $\operatorname{Tr}(A^{\top}A)=6$.
My approach
Assuming $A=\begin{bmatrix} a_{11}&a_{12} & a_{13}\\ a_{21} & a_{22} &a_{23} \\ a_{31} &a_{32} & a_{33} \end{bmatrix}$ and hence $A^{\top}=\begin{bmatrix} a_{11}&a_{21} & a_{31}\\ a_{12} & a_{22} &a_{32} \\ a_{13} &a_{23} & a_{33} \end{bmatrix}$
We have
$A^{\top}A=\begin{bmatrix} a_{11}^{2}+ a_{21}^{2}+ a_{31}^{2}&\lambda_{1} & \lambda_{2}\\ \lambda_{3}& a_{12}^{2}+ a_{22}^{2}+ a_{32}^{2}& \lambda_{4}\\ \lambda_{5}& \lambda_{6}& a_{13}^{2}+ a_{23}^{2}+ a_{33}^{2} \end{bmatrix}$
So the problem reduces to finding integers $a_{ij}$; $i,j=1,2,3$ such that
$\sum_{i=1}^{3}\sum_{j=1}^{3}a_{ij}^{2}=6$ which can be thought of as $\sum_{i=1}^{9}t_{i}=6$ where each $t_{i}$ is an integer and perfect square.
Hence we must partition 6 into 9 portions each of which is a perfect square,
Case 1;
9 $t_{i}s$= 6($t_{i}s$ have value $1$)+3($t_{i}s$ have value $0$)
number of matrices here would be $n_{1}=\frac{9 !}{6!3!}2^{6}$, since each permutation of the string of 6 1’s and 3 0’s corresponds to a value of $t_{i}$ and hence $a_{ij}^{2}$ and because for each value of $a_{ij}^{2}$=1, we can have $a_{ij}=\pm1$, hence the number of permutations is scaled up by $2^{6}$
Case 2
9 $t_{i}s$= 1($t_{i}$ has value $4$)+2($t_{i}s$ have value $1$)+6($t_{i}s$ have value $0$)
number of matrices here would be $n_{2}=\frac{9 !}{1!2!6!}2^{3}$, since each permutation of the string of 1 2’s and 2 1’s and 6 0’s corresponds to a value of $t_{i}$ and hence $a_{ij}^{2}$ and because for each value of $a_{ij}^{2}$=1, we can have $a_{ij}=\pm1$ and similarly for $a_{ij}^{2}$=4, hence the number of permutations is scaled up by $2^{3}$
No further partitions into perfect squares are possible
Thus the total number of matrices should be $N=n_{1}+n_{2}=\frac{7\cdot8\cdot9\cdot64}{6}+7\cdot8\cdot9\cdot4=7392$
Is the above reasoning correct for the number of such matrices ?
Please help me if anything is missing in above since I don’t have the answer for this in the exercise.
Your reasoning is entirely correct. The presentation can be made a bit clearer as follows:
Once you have reduced the problem to solving $\sum_{i=1}^9t_i^2=6$ over the integers, you can immediately note that $|t_i|\in\{0,1,2\}$ for all $i$. Let $n_k$ denote the number of indices $i$ with $|t_i|=k$. Then $$n_0+n_1+n_2=9\qquad\text{ and }\qquad n_0\cdot0^2+n_1\cdot1^2+n_2\cdot2^2=6.$$ From here it is clear that $n_2\leq1$. For $n_2=1$ you get $n_1=2$ and $n_0=6$, indeed yielding $$2^3\cdot\frac{9!}{1!2!6!}=2016,$$ such matrices. For $n_2=0$ you get $n_1=6$ and $n_0=3$, yielding $$2^6\cdot\frac{9!}{0!6!3!}=5376,$$ such matrices for a total of $7392$ matrices.