Regularized Least Squares Objective - Sufficient and Necessary Conditions for Unique Solution

1.2k Views Asked by At

Consider some form of Tikhonov regularization, where we seek to minimize the objective described by:

$\min_{x \in \mathbb{R}^n} ||Ax - b||^2_2 + \lambda||Dx||^2_2$

(with matrices $A, D$ and vectors $b, x$ of appropriate shape, and some scalar $\lambda >0$). I've read from a text that a unique solution exists for the above objective if and only if $ker(A) \cap ker(D) = \{0\}$, and I'm struggling with the "if" direction of the proof. In particular, I've derived the normal form of the solution, which much satisfy:

$(A^TA + \lambda D^TD)x = A^Tb$

and I've shown that if $z \in ker(A) \cup ker(D)$ for non-zero $z$, then if $x^*$ is a solution to the above normal form, then so is $x^* + z$. However, I'm not sure how to proceed with assuming that if $ker(A) \cap ker(D) = \{0\}$, then we must have a unique $x^*$ as the solution.

EDIT: In fact, is there a cleaner solution that does not require proving both conditionals in the biconditional separately like I've formulated?

2

There are 2 best solutions below

2
On BEST ANSWER

The Hessian of the convex quadratic objective is $H=A^TA + \lambda D^TD$, and so the solution is unique iff $\ker H$ is trivial.

It is straightforward to check that $Hx = 0$ iff $Ax=0 $ and $Dx=0$ and so $\ker H = \ker A \cap \ker D$.

Note that this is more obvious if we write $H = \begin{bmatrix} A^T & \sqrt{\lambda}D^T \end{bmatrix} \begin{bmatrix} A \\ \sqrt{\lambda}D \end{bmatrix}$.

0
On

Hint: Note that $$ \|Ax-b\|^2 + \lambda \|Dx\|^2 = \| \begin{bmatrix} A \\ \sqrt{\lambda} D \end{bmatrix} x - \begin{bmatrix} b \\ 0 \end{bmatrix}\|^2. $$