Inverse of the subtraction of Matrices

Question

I would like to compute $$(\mathbf{P}-\frac{1}{\sigma^2}\mathbf{I})^{-1},$$ which I already have $\mathbf{P}^{-1}$ and $\mathbf{I}$ is the identity matrix. It is not clear to me how to compute it for instance from Sherman-Morrison formula. I can write $$(\mathbf{P}-\frac{1}{\sigma^2}\mathbf{I})^{-1}=\mathbf{P}^{-1}+\mathbf{X}.$$ and then I have $$(\mathbf{P}-\frac{1}{\sigma^2}\mathbf{I})(\mathbf{P}^{-1}+\mathbf{X})=\mathbf{I}.$$ Doing a bit of math, I will get $$\mathbf{X}=\frac{\mathbf{P}^{-1}}{\sigma^2\mathbf{P}-\mathbf{I}}.$$ But computing the denominator is expensive. Any suggestion how to calculate it?