Continuity of Lasso solution with respect to the regularization parameter

Question

Consider the following real Lasso (or Basis Pursuit Denoising) problem:

$$\min_x \ \lvert\lvert Hx-y\lvert\lvert^2+\lambda \lvert x\lvert_1$$

Where $y\in R^m,x\in R^n,H\in R^{m\times n},\lambda\in R, \lambda \geq 0$, $H$ has full column-rank.

If we denote the problem's solution as a function of $\lambda$, namely, $x=x(\lambda)$, then, is $x(\lambda)$ continuous?

Answer 1

Yes, $x(\lambda)$ is continuous (even locally Lipschitz continuous!) under some conditions on the design matrix $H$ (namely, some restriction of $H^TH$ should be positive definite). To see this, you can apply the implicit function theorem of Clarke to the optimality condition:

$$x^\star \mbox{ solves the LASSO problem with parameter $\lambda$}\iff \mathrm{prox}_{\lambda|\cdot|_1}(x^\star-H^T(Hx^\star-y))=x^\star$$