Currently, I'm stuck to one of a statement in a paper. Following is a brief summary of the paper regarding my question. (although the topic of the paper is mainly statistics, the question purely relies on mathematics)
For $\lambda>0,\beta\in\mathbb R^p,n<p,X\in\mathbb R^{n\times p} ,\Sigma\in\mathbb R^{p\times p}$, let $B_{X,\Sigma} := \lambda^2\beta^T(S_X+\lambda I)^{-1}\Sigma(S_X+\lambda I)^{-1}\beta$ where $S_X= X^TX/n$. Under these facts, show the following inequality, where $C_1,C_2>0$ are constants: $$\begin{aligned} |B_{X_1,\Sigma_1}- B_{X_2,\Sigma_2}| &\le C_1\Vert S_{X_1}-S_{X_2}\Vert_{op} + C_2\Vert\Sigma_1-\Sigma_2 \Vert_{op} \\ &\le \frac{C_1}{n}(\Vert X_1\Vert_{op} + \Vert X_2\Vert_{op} )\cdot\Vert X_1-X_2\Vert_{op} + C_2\Vert\Sigma_1-\Sigma_2 \Vert_{op} \end{aligned}$$
The second inequality is straightforward by "adding and subtracting" some terms $X_1^TX_2$ and applying $\Vert A+B\Vert_{op} \le \Vert A \Vert_{op} +\Vert B \Vert_{op}$ and $\Vert AB \Vert_{op} \le \Vert A \Vert_{op}\Vert B \Vert_{op}$.
The thing is the first inequality. I tried to "add and subtract" several term between $B_{X_1,\Sigma_1}$ and $B_{X_2,\Sigma_2}$ such as $\lambda^2\beta^T(S_{X_1}+\lambda I)^{-1}\Sigma_2(S_{X_1}+\lambda I)^{-1}\beta$, but I'm not sure whether this kind of approach is correct and if so, how I can proceed from here. Especially, I need to extract the terms $S_{X_1}-S_{X_2}$ and $\Sigma_1-\Sigma_2$, but this isn't simple because of the inverse term $(S_X+\lambda I)^{-1}$.
Any help regarding this would be appreciated. Thank you.
To simplify notation, you are trying to bound an expression of the form $$ \lambda^2\big|\beta^TA_1^{-1}F_1A_1^{-1}\beta-\beta^TA_2^{-1}F_2A_2^{-1}\beta\big|, $$
where $F_j=\Sigma_j$ and $A_j=S_{X_j}+\lambda I$.
Note that $(S_{X_j}+\lambda I)\geq\lambda I$, so $\|(S_{X_j}+\lambda I)^{-1}\|\leq\lambda^{-1}$ and so $\|A_j^{-1}\beta\|\leq\lambda^{-1}\,\|\beta\|$. Also $\|\beta^T A_j^{-1}\|\leq\lambda^{-1}\,\|\beta\|$, $\def\abajo{\\[0.2cm]}$ and \begin{align} \|A_1^{-1}-A_2^{-1}\|&=\|A_2^{-1}(A_2-A_1)A_1^{-1}\|\leq\|A_2^{-1}\|\,\|A_1^{-1}\|\,\|A_2-A_1\|\abajo &\leq\lambda^{-2}\,\|A_2-A_1\|. \end{align}
Let $L=\big\|A_1^{-1}F_1A_1^{-1}-A_2^{-1}F_2A_2^{-1}\big\|$. You have \begin{align} L &\leq\big\|A_1^{-1}F_1A_1^{-1}-A_1^{-1}F_1A_2^{-1}\big\| +\big\|A_1^{-1}F_1A_2^{-1}-A_1^{-1}F_2A_2^{-1}\big\|\abajo &\qquad\qquad+\big\|A_1^{-1}F_2A_2^{-1}-A_2^{-1}F_2A_2^{-1}\big\|\abajo &=\big\|(A_1^{-1}F_1)(A_1^{-1}-A_2^{-1})\big\| +\big\|A_1^{-1}(F_1-F_2)A_2^{-1}\big|+\big\|(A_1^{-1}-A_2^{-1})F_2A_2^{-1}\big\|\abajo &\leq\big\|A_1^{-1}F_1\|\,\|A_1^{-1}-A_2^{-1}\big\| +\lambda^{-2}\big\|F_1-F_2\big\|+\lambda^{-2}\big\|F_2A_2^{-1}\big\|\,\|A_2-A_1\|\abajo &\leq \lambda^{-3}(\|F_1\|+\|F_2\|)\,\|A_2-A_1\|+\lambda^{-2}\,\|F_2-F_1\|. \end{align} Then \begin{align} |B_{X_1,\Sigma_1}- B_{X_2,\Sigma_2}| &=\lambda^2\big|\beta^T(A_1^{-1}F_1A_1^{-1}-A_2^{-1}F_2A_2^{-1})\beta\big|\abajo &\leq\lambda^2\|\beta\|^2\,\big\|A_1^{-1}F_1A_1^{-1}-A_2^{-1}F_2A_2^{-1}\big\|\abajo &\leq\lambda^{-1}\|\beta\|^2(\|F_1\|+\|F_2\|)\,\|A_2-A_1\|+\|\beta\|^2\,\|F_2-F_1\|. \end{align}