I've been reading this code and I found that the number of non-zero eigenvalues of the estimated covariance is equal to $x_i^T \Sigma^{-1} x_i$. I want to know how to arrive at this result.
Some background:
- $x_i$ is a real column vector with dimension $d$ (one sample)
- $X = [x_1, x_2, ..., x_n]$ with shape $d$x$n$ (all the samples)
I want to prove that:
$$\sum_{i=0}^{i=n} x_i^T \Sigma^{-1} x_i = n.len(s)$$
being $len(s)$ the number of non-zero singular values* of $\Sigma$, that is defined as
$$\Sigma = \frac{1}{n} \sum_{j=1}^{j=n} x_j x_j^T$$
If necessary, mean can be considered $0$
*Not necessarily mathematical $0$, this can also mean "not too small values" .
Actually non-zero is "non-negligible" depending on a threshold defined as the largest singular value times the square root of the machine epsilon.
In Python: s[0] * np.sqrt(np.finfo(np.float).eps) being s the singular values in descending order (see the code)
Assuming that the sample $\{x_i\}$ spans the whole space $\mathbb{R}^d$, the value is always $nd$. Here is the proof:
Using your notation for $X$, we know $\Sigma = \frac{1}{n} XX^T$ and therefore $\Sigma^{-1} = n \left(XX^T \right)^{-1}$. Then $$ x_i^T \Sigma^{-1} x_i = tr(x_i^T \Sigma^{-1} x_i) = tr(\Sigma^{-1} x_i x_i^T) $$
and therefore $$ \sum_{i = 1}^n x_i^T \Sigma^{-1} x_i = tr(\Sigma^{-1} \sum_{i=1}^n x_i x_i^T) = tr(\Sigma^{-1} (XX^T)) = n tr(I_d) = nd $$ where $I_d$ is the $d$-dim identity matrix.