I believe that $A^TA$ is a key matrix structure because of its connection to variance-covariance matrices, which are used in data analysis (e.g. linear regression or principal component analysis). There are interesting properties to $A^TA$, which I think include symmetry and positive semi-definiteness, and real eigenvalues. Just to share my approximate degree of familiarity with the topic.
However, I am missing at least a rung in the ladder when attempting to understand this answer in CV, which revolves around the expression:
$$X'X = V D V', \quad D = \begin{bmatrix} d_1 & & \\ & \ddots & \\ & & d_p \end{bmatrix}, d_i \geq 0$$
In particular I would appreciate being tipped-off as to why the following statement is obvious:
"Now since matrix inversion corresponds to inversion of the eigenvalues, the OLS estimator requires $(X′X)^{-1}=VD^{−1}V′$ (note that $V′=V^{−1}$). Obviously this only works if all eigenvalues are strictly greater than zero, $d_i>0$."
May be some Linear algebra facts/results will make it clearer. From the equality $X'X = VDV'$, where $V$ is orthogonal matrix, hence $V'=V^{-1}$, you deduce that $X'X$ is similar to $D=diag(\lambda_1,...,\lambda_2)$. As such, they have the same eigenvalues. So, because the OLS estimator is defined by $(X'X)^{-1}X'y$, you need to invert the $(X'X)$ and moreover its inverse have to be positive definite matrix in order to minimize the square errors. This stems from the fact that $(X'X)^{-1}$ is the Hessian matrix of the $S(\beta)=(y-X\beta)'(y-X\beta)$. Thus, for $(X'X)$ to be invertible it has to be full-ranked matrix, namely all eigenvalues have to differ from $0$, and for the positive definite property it is required that each its eigenvalue will be strictly positive, i.e. $\lambda_i >0$, $\forall i$.