Maximising the variance in PCA - Trace Proof

Question

Quite urgent request for a proof here as my professor has not run through it.

Scenario is the PCA problem where we are trying to maximise the variance in the latent space. I want to prove the following $\sum_k^d W_k^TS_tW_k = Trace[W^TS_tW]$ where $S_t$ is the covariance matrix of dimension $F \times F$. $W_k$ is the $k^{th}$ eigenvector of $S_t$ $\in F \times 1$ and $W = [W_1, W_2, .., W_k]$.

I sort of know the intuition behind it, the eigenvectors are orthogonal and thus $W_k^TW_j$ where $k \neq j = 0$. I am having issue with the formal proof.

I have tried the following

$\sum_k^d \big(\sum_i^F W_{k,i}St_{i,j} W_{j,k}\big)$ and stated that if $i \neq j, W_{j,k}W_{k,i} = 0$

Does anyone have a better proof for this please? or provide a hint as to where I am going wrong

Answer 1

The trace of $W^\top S_t W$ is the sum of is diagonal entries. Show that the $k$th diagonal entry is $W_k^\top S_t W_k$. (Think about how to compute the matrix multiplication $W^\top S_t W$.)

Answer 2

With help of the trace identities $a^Ta = \text{tr}(aa^T)$ and $\text{tr}(ABC) = \text{tr}(BCA)$ for vector $a$ and suitable martrices $A,B,C$ we get

$$\text{tr}\left(W^TS_tW\right) = \text{tr}\left(S_tWW^T\right) = \sum_{k=1}^d\text{tr}\left(S_tW_kW_k^T\right) = \sum_{k=1}^d\left(S_tW_k\right)^TW_k = \sum_{k=1}^dW_k^TS_tW_k. $$