On SVD, PCA and optimization

Question

Let's say we got a data matrix $X_{n,p}$ wich represent $n$ person with $p$ feature. We can represent each person in an $R^n$ space, then we get a data cloud.

One goal of PCA is to find the "best approximation" of the data cloud in a $k$ dimensional subspace of $R^n$. This best approximation is introduced in various geometric way:

1. We look for a projection $P_{E_k}$ such that the sum of the distance between the "projected person" would be maximum.

2. Similar to "least square"method we look for an "orthogonal projection" which would minimize the sum of distance between the points and their projected.

3. We look for the maximum "inertie" or Variance of the projected cloud which geometrically give the total distance between the projected cloud and his bartcenter.

4. Finally a non geometric approach lead us to look for the "best" low rank (rank $k $) approximation of $X $ (cf.

Eckart-Young theorem. My questions are the following (please don't miss the drawing at the end)

A) Could you prove the equivalence between all theses statement?

B) In 1) we consider all the projection whereas in 2) we focused on orthogonal projection. How is that possible ? How do we finally get that the "best" projection is an orthogonal one?

PS: All my question raised from a really interesting presentation wich suffer to prove nothing: http://bertrand.michel.perso.math.cnrs.fr/Enseignements/5MS04/ACP-AFC-ACM.pdf

---

---

Could you prove the equivalence of the 4 point of view ?

Thanks

Answer 1

PCA is "Best" in the sense it preserves most of the energy of the data.
The best way to answer your questions is to analyze the Principal Component Analysis (PCA) using the Singular Value Decomposition (SVD).

Then you'll be able to understand what PCA really does.
Basically, if look at your data as a matrix, the PCA (Using the SVD) is creating the best low rank approximation of the data.

Best in the sense it preserves most f the energy for given rank where the energy is measured by the Singular Values of the data.

You may have a look in my Singular Value Decomposition (SVD) Presentation.
Specifically pages 35-43 and 80-86.