According to the research paper 'An alternative approach for solving the problem of close to singular covariance matrices in modern portfolio theory', if a matrix ($ \Sigma\ $) is close to singularity, the inverted covariance matrix ($ \Sigma^{-1}\ $) contains high estimation error.
... the sample covariance matrix approaches singularity. Both problems lead to uncertainties in form of estimation error. ....
I have tried hard to figure out why the inverted covariance matrix ($ \Sigma^{-1}\ $) high estimation error (=poor quality) when the covariance matrix ($ \Sigma\ $)is close to singularity but cannot figure out why. Any explanation in plain English would be very much appreciated
Let $X$ be the $n\times p$ matrix of data ($n$ observations and $p$ features). If the data is centered, the estimated covariance matrix is $\hat{\Sigma}=n^{-1}X^{T}X$. This matrix, as the original $\Sigma$, is non-negative definite (all its eigenvalues are $\geq 0$). If $\lambda_1\geq \cdots\geq \lambda_p\geq 0$ are the eigenvalues of $\hat{\Sigma}$, you can diagonalize it as $$ \hat{\Sigma}=P\text{diag}(\lambda_1,...,\lambda_p)P^{T}, $$ for some orthogonal matrix $P$. And its inverse is of course $$ \hat{\Sigma}^{-1}=P\text{diag}(\lambda_1^{-1},...,\lambda_p^{-1})P^{T}. $$ If $X$ has rank $< p$ (for example if $n<p$), then there exist some $\lambda_i=0$ and you can not compute $\hat{\Sigma}^{-1}$. So if you have some $\lambda_i\approx 0$, the estimation of $\Sigma^{-1}$, i.e, $\hat{\Sigma}^{-1}$ will be unstable (some of its elements will explode), in particular $\text{trace}(\hat{\Sigma}^{-1})$ will approach infinity.
I recommend you the book "Large Sample Covariance Matrices and High‑Dimensional Data Analysis" by Yao, et. al. for more on the estimation of $\Sigma$ when $p>n$.