The covariance matrix of any data X -> N * D (N samples and D dimension ) would be Cov(X) = E[(X - E[X])(X - E[X])T]. Let's Assume (X - E[X])=Y. Thus Cov(X) = E[YYT]. Now from SVD we know that U and V are just eigenvectors of ATA and AAT respectively. Thus, we can just use Eigen decomposition of YYT as it is symmetric.
The confusion I face is how do you interpret this Cov(X) = Eigendecomposition(YYT) = PDP-1. From my point of view it is just a factorization of linear transformation.
How can we conclude that which dimension would have highest covariance from this factorization? I am having hard time interpreting it as anything but transformation, meaning if I multiply a vector with PDP-1, the vector will get transformed to a space where eigen vectors are basis. In this case they are orthonormal and hence its just rotation and then scaled and then put back to original space.
All I can view it as a factorization which tells us about the Cov(X) as a transformation rather than Cov(X) itself.
According to Wikipedia Principal Component Analyis comes in two ways:
As a basis transformation (diagonalization) of the covariance matrix: $C=PDP^\top$
As a singular value decomposition of the (transposed) data matrix: $X^\top=U\Sigma W^\top$ (assuming we have already subtracted the mean from $X$)
Since $C=XX^\top=W\Sigma^2 W^\top$ and since $\Sigma$ is a diagonal $(d\times d)$-matrix it is clear that $\Sigma^2=D$ and $W=P\,.$ In other words, both approaches lead to the same diagonalization of the covariance matrix.
From basic linear algebra it is clear that the diagonal matrix $D$ contains the eigenvalues and the columns of $P$ are the eigenvectors. It is also clear that any permutation of those columns will just permute the diagonal elements of $D\,.$
We cannot conclude 'which dimension would have the highest covariance' but we know always which eigenvector corresponds to the highest, second highest, and so on, eigenvalue. That's all that counts.