Why covariance matrix estimation error is high when the matrix is close to singularity?

566 Views Asked by At

According to the research paper 'An alternative approach for solving the problem of close to singular covariance matrices in modern portfolio theory', if a matrix ($ \Sigma\ $) is close to singularity, the inverted covariance matrix ($ \Sigma^{-1}\ $) contains high estimation error.

... the sample covariance matrix approaches singularity. Both problems lead to uncertainties in form of estimation error. ....

I have tried hard to figure out why the inverted covariance matrix ($ \Sigma^{-1}\ $) high estimation error (=poor quality) when the covariance matrix ($ \Sigma\ $)is close to singularity but cannot figure out why. Any explanation in plain English would be very much appreciated

2

There are 2 best solutions below

2
On BEST ANSWER

Let $X$ be the $n\times p$ matrix of data ($n$ observations and $p$ features). If the data is centered, the estimated covariance matrix is $\hat{\Sigma}=n^{-1}X^{T}X$. This matrix, as the original $\Sigma$, is non-negative definite (all its eigenvalues are $\geq 0$). If $\lambda_1\geq \cdots\geq \lambda_p\geq 0$ are the eigenvalues of $\hat{\Sigma}$, you can diagonalize it as $$ \hat{\Sigma}=P\text{diag}(\lambda_1,...,\lambda_p)P^{T}, $$ for some orthogonal matrix $P$. And its inverse is of course $$ \hat{\Sigma}^{-1}=P\text{diag}(\lambda_1^{-1},...,\lambda_p^{-1})P^{T}. $$ If $X$ has rank $< p$ (for example if $n<p$), then there exist some $\lambda_i=0$ and you can not compute $\hat{\Sigma}^{-1}$. So if you have some $\lambda_i\approx 0$, the estimation of $\Sigma^{-1}$, i.e, $\hat{\Sigma}^{-1}$ will be unstable (some of its elements will explode), in particular $\text{trace}(\hat{\Sigma}^{-1})$ will approach infinity.

I recommend you the book "Large Sample Covariance Matrices and High‑Dimensional Data Analysis" by Yao, et. al. for more on the estimation of $\Sigma$ when $p>n$.

12
On

When $\Sigma$ is the covariance matrix and $w$ is the vector of investments, then $w^T\Sigma w$ is the variance of the portfolio. Suppose $\Sigma$ is almost singular, then there is an eigenvector $v$ for which the eigenvalue $\lambda$ is close to $0$. Taking $w=v$, you get variance $\lambda w^Tw$. Being able to make an investment with near $0$ variance can be an arbitrage opportunity depending on the context.