Let:
\begin{equation} \mathbf{M}= \begin{bmatrix} \mathbf{A}_{11} & \mathbf{D}_{12} & \mathbf{D}_{13} \\ \mathbf{D}_{12} & \mathbf{A}_{22} & \mathbf{D}_{23} \\ \mathbf{D}_{13} & \mathbf{D}_{23} & \mathbf{A}_{33} \end{bmatrix} \end{equation}
where $\mathbf{A}_{ii}$ is a full matrix and $\mathbf{D}_{ij}$ is a diagonal matrix, with $i,j= 1,2,3$.
How to efficiently compute (computationally speaking) $\mathbf{M}^{-1}$?
As far as I know, there is no trick that will allow you to utilized this structure under the stated general assumptions.
You question implies that $M$ is nonsingular and that you have enough memory to store the inverse matrix $M^{-1}$ explicitly.
I will assume that that the matrix has an LU factorization. In this case, you do a partitioned LU factorization, but fill-in will destroy the diagonal structure of the off diagonal blocks.
I will show how to derive a partitioned $LU$ factorization. Let $$A = \begin{bmatrix} A_{11} & A_{12} \\ A_{21} & A_{22} \end{bmatrix}$$ be a block matrix. We want $A = LU$ where $$ L = \begin{bmatrix} L_{11} & 0 \\ L_{21} & L_{22} \end{bmatrix}, \quad U = \begin{bmatrix} U_{11} & U_{12} \\ 0 & U_{22} \end{bmatrix}.$$ By explicitly multiplying $L$ and $U$ together and comparing with $A$ we deduce $$ L_{11} U_{11} = A_{11}, \: L_{21} U_{11} = A_{21}, \: L_{11} U_{12} = A_{12}, \: L_{21} U_{12} + L_{22} U_{22} = A_{22}.$$ From this an algorithm emerges,