Trying to understand how to invert a covariance matrix under a factor model decomposition. Looks like there's an identity called the Woodbury matrix identity here.
My question is if I have a factor model decomposed covariance matrix $Σ = B Cov(F) B' + Cov(e)$ where $B$ are the factor exposures, $F$ are the factor returns and $e$ are the residual returns, and I'm trying to compute $Σ^{-1} * \mathbf{w}$ such that $B * w = 0$ (e.g. all the aggregate factor exposures sum to 0) is there a way to simplify the $Σ^{-1}$ expression using the Woodbury matrix identity? Wondering if terms drop out such that you are only concerned with the inverse of the residual covariance matrix or something like that