Use PCA to calculate the direction vector of the given data set.

Question

I'm trying to work through a simple example where I'm given three points in a data set, $(0,2), (1,-1), (-1,-1)$, and using PCA I have to find a direction, so that the quadratic distance of the data points to the associated straight line of the origin is minimised. I compared what I learned from class and a lot of YouTube videos and here's what I have so far.

1. Find the mean of the data points. $$\begin{bmatrix}x_1\\x_2\end{bmatrix}=\begin{bmatrix}0\\2\end{bmatrix}, \begin{bmatrix}1\\-1\end{bmatrix}, \begin{bmatrix}-1\\-1\end{bmatrix}= \begin{bmatrix}0\\0\end{bmatrix}$$ where $\begin{bmatrix}0\\0\end{bmatrix}$ is the mean of $\begin{bmatrix}x_1\\x_2\end{bmatrix}$.

2. Then I tried to find the covariance matrix by first calculating $(x-\mu )$, which returns the original data points, since the mean is $\begin{bmatrix}0\\0\end{bmatrix}$.

3. I then found the covariance matrix by using the formula $(x-\mu)(x-\mu)^T$, which returns the following matrices:

$$\begin{bmatrix}0 & 0\\0 & 4\end{bmatrix}, \begin{bmatrix}1 & -1\\-1 & 1\end{bmatrix},\begin{bmatrix}1&1\\1&1\end{bmatrix}$$

4. Their sum is $\begin{bmatrix}2&0\\0&6\end{bmatrix}$, which means that the covariance matrix is $\frac{1}{2} \begin{bmatrix}2&0\\0&6\end{bmatrix}=\begin{bmatrix}1&0\\0&3\end{bmatrix}$
5. I then found the Eigen values, using the formula $Ax = \lambda x$, which gives $\begin{bmatrix}1&0\\0&3\end{bmatrix}x = \lambda x$. Using the determinant I got $\begin{vmatrix}1-\lambda&0\\0&3-\lambda\end{vmatrix}=0 \implies (-x+1)(-x+3)=0 \implies x_1=1, x_2=3$
6. Choosing the larger $x$ value, i.e. $x_2=3$, and put that back into the equation $Ax= \lambda x \implies \begin{bmatrix}1&0\\0&3\end{bmatrix} \begin{bmatrix}x_1\\x_2\end{bmatrix} = 3 \begin{bmatrix}x_1\\x_2\end{bmatrix}$, which gave me the result $x_1=0$ but I don't get a solution for $x_2$ because I get $0 \cdot x_1 + 3 \cdot x_2=3x_2 \implies 0 = 0$

Since I have to get a solution, it clearly means that I made a major mistake somewhere but I've been trying to look at other examples and they mostly do it in a similar manner and get proper solutions. Therefore, I'd like to ask what am I not getting correct here?

Answer 1

It means $x_2$ can take any values.

In particular, $\begin{bmatrix} 0 \\ 1 \end{bmatrix}$ is an eigenvector with corresponding eigenvalue $3$. Any non-zero multiple of it is also an eigenvector.