Let $X=(X_1,...,X_p)$ be a random vector with $\mathbb{E}(X)=\mu$ and covariance matrix $\text{Cov}(X)=\Sigma$. Define $Y=(Y_1,...,Y_p)^T=\mathbb{E}^TX$, where $E$ is a $p\times p$ matrix with columns equal to the eigenvectors of $\Sigma$. Show that $$\frac{\sum^k_{i=1}\text{Var}(Y_i)}{\sum^p_{i=1}\text{Var}(X_i)}=\frac{\sum^k_{i=1}\lambda_i}{\sum^p_{i=1}\lambda_i}$$
where $\lambda_1,...,\lambda_p$ are the eigenvalues of $\Sigma$.
Here is how I tackle the problem:
To show that $\sum^p_{i=1}\text{Var}(X_i)=\sum^p_{i=1}\lambda_i$:
We have that $$\sum^p_{i=1}\text{Var}(X_i)=Tr(Cov(X))=Tr(\Sigma)=Tr(E\Lambda E^T)=Tr(E^TE\Lambda)=Tr(\Lambda)=\sum^p_{i=1}\lambda_i$$
To show that $\sum^k_{i=1}\text{Var}(Y_i)=\sum^k_{i=1}\lambda_i$:
We have that $\text{Var}(Y_i)=\text{Var}(e_i^TX)=e_i^T\text{Cov}(X)e_i=e_i^T\Sigma e_i$.
So, $$\sum^k_{i=1}e_i^T\Sigma e_i=\sum^k_{i=1}e^T_iE\Lambda E^T e_i$$
We can see that $e_i^TE=s_i$, where $s_i$ is a p-dimensional row vector with entries 0 except for the $i$-th entry. This is because the columns of the matrix $E$ are orthonormal by definition. Furthermore, $E^Te_i=s_i^T$.
So, we have that $s_i\Lambda=l_i$, where $l_i$ is a p-dimension row vector, where the $i$-th entry is equal to $\lambda_i$. Hence, we get that $e_i^TE\Lambda E^Te_i=l_is_i^T=\lambda_i$.
Hence, we get that $\sum^k_{i=1}e_i^TE\Lambda E^Te_i=\sum^k_{i=1}\lambda_i$.
Would the way I approached this be correct?
Seems right to me.But let me give you this:
Let a a matrix $X \in \mathbb{R}^{n \times p}$ where
- $n:$ the # of rows (observations) and - $p:$ the # of columns (variables)
$$S=P \Lambda P^{T}$$ , where $P$ is the matrix of eigenvectors $\Lambda$ the diagonal matrix of eigenvalues $\lambda_{i}$ in decreasing order.The eigenvectors are the principal axis or the principal directions of the data (observations).
An orthogonal transformation of the standardized matrix $\hat{X}$ is: $$\hat{Y} = \hat{X} P $$
From the above we can see that the matrix $P$ is the same as $U$ of the $\textbf{SVD}$ of the original $X$
We have 2 standardized matrices $\hat{X}$ and $\hat{Y}$ where the $\textbf{SVD}$ of each matrix give us:
$$ \hat{X} = Q_{x} \begin{bmatrix} \Sigma & 0\\ 0& 0\\ \end{bmatrix} P_{x}^{T} \: \: \: \: , \: \: \: \: \hat{Y} = Q_{y} \begin{bmatrix} \Psi & 0\\ 0& 0\\ \end{bmatrix} P_{y}^{T} $$
Therefore the the PCA problem is now a problem of calculation of correlation these two matrices.We are searching a transformation matrix $Α$ of the columns of of $\hat{Y}$ such that : $$Ζ = \hat{Y}A.$$ The Variance - Covariance matrix of $\hat{X},\hat{Y}$ is given by: $$ Cor(X) = \dfrac{Z^{T}Z}{n-1}$$ Letting $$C = \Psi \hat{P}_{y}A$$gives us: $$Z = Q_{y} \begin{bmatrix} \Psi \hat{P}_{y}A \\ 0 \\ \end{bmatrix}$$ such that $Z^{T}Z = C^{T}C$
and therefore if the columns of $Z$ are orthonormal co is $C$
In addition the total variance is being explained from $$\dfrac{ \lambda_{i}}{\sum_{i=1}^{r}\lambda } .$$