For a transformation $A \in \mathbb{R}^{n\times m}$ what exactly is the geometric interpretation of the transformation $A^TA$. If I understand it correctly the entries of $A^TA$ are the inner products or the columns of $A$ but how exactly should I interpret this geometrically as a linear transformation? And why is $A^TA$ often loosely called squaring the matrix, how does having the pairwise inner products (which normally are interpreted as projecting one vector on the other) yield us something close to a matrix squared?
One thing I've noticed is that using the SVD for a real matrix $A$ we get $A = UDV^T \leftrightarrow A^TA=VD^TU^TUDV^T=VD^TDV^T$ where $U$ and $V$ are orthogonal, but how does changing basis to $V$ and scaling by the eigenvalues squared relate to the concept above?
$A^TA$ is a square matrix. As you said, its entries are the inner products of the columns of A.
The determinant of $A^TA$ will therefore be the Grammian of columns of $A$ and will be $>=0$ always. It is zero only when the columns of A are linearly dependent. To get an intuitive idea, you may think of the Grammian as the square of the volume of the parallelepiped (in $R^n$) formed by the vectors that form the columns of $A$.
If the vectors are linearly dependent, then of course all of the vectors can be written as a linear combination of each other and thus this volume is zero (It is easy to imagine this in $R^3$).
If they are orthogonal, then they are also linearly independent, and this volume is simply the product of the norms of the vectors, which agrees with the fact that $Gram(v_1, v_2, . ., v_k) \leq ||v_1||^2 . . ||v_k||^2$ where the equality holds under two conditions: