Covariance matrix decomposition - is $A^T\Sigma' A = A\Sigma' A^T$?

Question

Is $\Sigma = A^T\Sigma' A$ the same as $\Sigma = A\Sigma' A^T$ where $\Sigma$ is a symmetrical matrix and $\Sigma'$ is a diagonal matrix? If so, how can I prove this?

My textbook explains that a multivariate normal distribution can be written in terms of a diagonalised form of covariance $\Sigma$, using $\Sigma = A^T\Sigma' A$, by the process of singular value decomposition. However, other online sources explaining covariance decomposition mainly use $\Sigma = A^T\Sigma' A$ where it is described as an eigendecomposition. I'm confused as to whether they are the same thing.

Answer 1

They are different in the sense that you can't get from $A^T\Sigma' A$ to $A \Sigma' A^T$ (i.e. using the same matrix, $A$, in both forms) but they are the same in the sense that if $B = A^T$ then $A^T\Sigma' A = B \Sigma' B^T$.

In general, if $\Sigma$ is a symmetric matrix, then we can choose a matrix $A$ consisting of eigenvectors of $\Sigma$ (as columns) such that $A^{-1} = A^T$ (i.e. $AA^T = I$). Then to diagonalize $\Sigma$, we can write $\Sigma = A \Sigma' A^T$, where $\Sigma'$ is diagonal.

Alternatively, if we take $B = A^T$ then we have $B\Sigma B^T = \Sigma'$ (swapping the roles of $\Sigma$ and $\Sigma'$) and so $\Sigma = B^T \Sigma' B$ instead. Here the matrix $B$ has eigenvectors of $\Sigma$ as rows.

Answer 2

Those forms are equivalent. The first matrix in the product of three is a transpose of the third - it is up to you to decide which one you will call $A$ and which one you'll call $A^T$.

Answer 3

While it is true that the forms are equivalent (just take $A$ in place of $A^\top$) they are not equal which is probably what you are confused about.
To see that they are not equal just take some arbitrary matrices and multiply out, for example:

$$ A = \begin{pmatrix} 0 & 1 \\ 0 & 0 \end{pmatrix} \qquad \Sigma' = \begin{pmatrix} 1 & 0 \\ 0 & 2 \end{pmatrix} $$ then $$ A \Sigma' A^{\top} = \begin{pmatrix} 2 & 0 \\ 0 & 0 \end{pmatrix} $$ but $$ A^{\top} \Sigma' A = \begin{pmatrix} 0 & 0 \\ 0 & 1 \end{pmatrix} $$