Regularization of a Matrix Using a Diagonal Matrix

Question

For a $n\times n$ matrix $A$ it is well known that $A + \lambda I$ for suffiecientely large $\lambda >0$ makes $A$ positive definite. The proof is straightforward by looking at the characteristic polynomial. I wonder what happens if we look at a diagonal matrix $\Lambda$ with possibly distinct diagonal elements? Is the conclusion still valid that $A + \Lambda$ becomes positive definite for large enough $\Lambda$?

Background: I have implemented a code that performs regularization using the former approach. By experimental studies I found that using different values for regularization improves my results. So I was wonderung whether this also backed by theory.

Answer 1

The conclusion is still correct, as \begin{align*} \Lambda&=\text{diag}\left(\lambda_{1},\lambda_{2},\dots,\lambda_{n}\right)\\ &=\lambda_{\text{min}}I+\Xi\\ \Xi&=\text{diag}\left(\lambda_{1}-\lambda_{\text{min}}, \lambda_{2}-\lambda_{\text{min}},\dots,\lambda_{n}-\lambda_{\text{min}}\right)\\ \end{align*} Here $\lambda_{\text{min}}=\min_{1\leq j\leq n}\lambda_{j}$. For large enough $\lambda_{\text{min}}$, we have that $A+\lambda_{\text{min}}I$ is positive definite, that is for any nonzero vector $u$, we have \begin{align*} u^{\top}\left(A+\lambda_{\text{min}}I\right)u>0 \end{align*}

If $u$ is nonzero then

\begin{align*} u^{\top}\Xi u&=\sum_{j}u_{j}^{2}\left(\lambda_{j}-\lambda_{\text{min}}\right)\geq 0 \end{align*}

using this we have for every nonzero vector $u$

\begin{align*} u^{\top}\left(A+\Lambda\right)u&=u^{\top}\left(A+\lambda_{\text{min}}I+\Xi\right)u>0 \end{align*}

thus $A+\Lambda$ is guaranteed to be positive definite, provided that the minimum entry of $\Lambda$ is sufficiently large.

Answer 2

By definition $A + \lambda I$ is positive definite if, for every vector

$\vec x \ne 0, \tag 1$

we have

$\vec x^\bot(A + \lambda I) \vec x > 0; \tag{1.5}$

since $\vec x^\bot(A + \lambda I) \vec x= \vec x^\bot A \vec x + \vec x^\bot (\lambda \vec x) = \vec x^\bot A \vec x + \lambda \vec x^T \vec x, \tag 2$

clearly (1.5) binds for $\lambda > 0$ sufficiently large, since

$\exists M > 0, \; \vert \vec x^\bot A \vec x \vert < M \vec x^\bot \vec x; \tag 3$

we need only choose

$\lambda > M. \tag 4$

If now

$\Lambda = \text{diag}(\lambda_1, \lambda_2, \ldots, \lambda_n), \tag 5$

then

$\vec x^\bot (A + \Lambda) \vec x = \vec x^\bot A \vec x + \vec x^\bot \Lambda \vec x; \tag 6$

with

$\vec x = (x_1, x_2, \ldots, x_n)^\bot, \tag 7$

we have

$\Lambda \vec x = (\lambda_1 x_1, \lambda_2 x_2, \ldots, \lambda_n x_n)^\bot, \tag 8$

whence

$\vec x^\bot \Lambda \vec x = \displaystyle \sum_1^n \lambda_i x_i^2; \tag 9$

choosing each $\lambda_i > \lambda$ yields

$\vec x^\bot \Lambda \vec x = \displaystyle \sum_1^n \lambda_i x_i^2 > \lambda \sum_1^n x_i^2 = \lambda \vec x^\bot \vec x, \tag{10}$

from which in light of (1.5) shows that $M + \Lambda$ is positive definite.