Given are an $m$-dimensional vector-space $V$ and a base $\mathscr{B}$ of $m$ vectors $\mathbf{e}_i \in V$.
Given $m$ vectors $\mathbf{a}_j$ of an $n$-dimensional vector-space $W$, one can construct a unique linear transform $T:V\to W$ such that $T(\mathbf{e}_j)=\mathbf{a}_j$.
Given a base $\mathscr{D}$ of $n$ vectors $\mathbf{e'}_i\in W$, $\mathbf{a}_j$ can be written as $a_j^i\mathbf{e'}_i$.
Then, the linear transform $T$ can be uniquely represented by the $(n\times m)$-matrix $$A=\begin{bmatrix}a_1^1&a_2^1&\cdots&a_m^1\\a_1^2&a_2^2&\cdots&a_m^2\\\vdots&\vdots&\ddots&\vdots\\a_1^n&a_2^n&\cdots&a_m^n\end{bmatrix},$$ such that $\forall \mathbf{v}=v^j\mathbf{e}_j, T(\mathbf{v})=v^ja_j^i\mathbf{e'}_i$.
I will come to my point: $(A^TA)_{ij}=a_i^ka_j^k$. Given the metric $g_{ij}=\delta_{ij}$, we get that $(A^TA)_{ij}=\mathbf{a}_i\cdot\mathbf{a}_j$, and to make it clearer, in matrix form:$$A^TA=\begin{bmatrix}\mathbf{a}_1\cdot\mathbf{a}_1&\mathbf{a}_1\cdot\mathbf{a}_2&\cdots&\mathbf{a}_1\cdot\mathbf{a}_m\\\mathbf{a}_2\cdot\mathbf{a}_1&\mathbf{a}_2\cdot\mathbf{a}_2&\cdots&\mathbf{a}_2\cdot\mathbf{a}_m\\\vdots&\vdots&\ddots&\vdots\\\mathbf{a}_m\cdot\mathbf{a}_1&\mathbf{a}_m\cdot\mathbf{a}_2&\cdots&\mathbf{a}_m\cdot\mathbf{a}_m\end{bmatrix}.$$
What does that mean?
It looks like a broken metric tensor with basis $\{\mathbf{a}_i\}$ but in $V$ instead of $W$ because of the $(m\times m)$-dimension matrix.
EDIT:
It makes at least some sense. $A$ is the change of basis matrix, and since $g$ is covariant, one can write $$\begin{aligned}g_{kl}&=a_k^ia_l^jg_{ij}\\&=a_k^ia_l^j\delta_{ij}\\&=a_k^ja_l^j\\&=(A^TA)_{kl}.\end{aligned}$$
But I don't find it coherent enough since I don't know what the metric tensor defined in $V\times V$ would mean represented in not really a basis of vectors in $W$...