Solving a regularized linear least squares problem using only matrix-vector products $Ax$ and $A^Tx$

74 Views Asked by At

I'm working on an application that involves solving a regularized linear least squares problem of the form

\begin{align} \min_x \frac{1}{2} \| y - Ax\|_2^2 + \frac{\mu}{2} \| x - b \|_2^2, \tag{$\star$} \end{align}

where $\mu>0$ is given. Naturally, $(\star)$ could be solved analytically with

\begin{align} \hat{x} &= (A^T A + \mu I)^{-1} (A^T y + \mu b), \end{align}

but unfortunately, I don't have direct access to $A$, since its construction is very complicated and very demanding in terms of memory consumption. Instead of that, I have a black-box algorithmic procedure that computes the matrix-vector products $Ax$ and $A^Tx$ very efficiently. What are some examples of simple iterative procedures I could use to solve $(\star)$ at least approximately?