Proximal operator inequality under strong convexity assumption

225 Views Asked by At

Define the prox operator associated to a closed, convex, proper function $f$, $\operatorname{prox}_f(x)$ to be

$$\operatorname{prox}_f(x) = \arg\min\limits_{u}\left\{f(u) + \frac{1}{2}\|x-u\|^2\right\}$$

We have the following inequality given $y = \operatorname{prox}(x)$, for each $u$,

$$f(y) - f(u) +\frac{1}{2}\|y-u\|^2 + \frac{1}{2}\|y-x\|^2\leq \frac{1}{2}\|u-x\|^2$$

This can be derived using the definition of the prox operator and the fact that $f(\cdot) + \frac{1}{2}\|\cdot-x\|^2$ is strongly convex with respect to $\frac{1}{2}\|\cdot-x\|^2$. What if $f$ itself is strongly convex with respect to $\frac{1}{2}\|\cdot\|^2$? How does the inequality improve?

To prove the inequality above we do the following. Let $y=prox(x)$, then

$$0\in \partial\left(f(\cdot) + \frac{1}{2}\|\cdot-x\|^2\right)(y)$$

We have that $g(u) = f(u) + \frac{1}{2}\|u-x\|^2$ is strongly convex w.r.t. $\frac{1}{2}\|u-x\|^2$ since $f(u) + \frac{1}{2}\|u-x\|^2 - \frac{1}{2}\|u-x\|^2 = f(u)$ is convex. By this strong convexity of $g(u)$ we have

$$g(u) - g(u^*) \geq \frac{1}{2}\|u-u^*\|^2$$

where $u^*$ is the minimizer of $g$. Since $y$ is exactly $u^*$, we get, for any u

$$g(u) - g(y) \geq \frac{1}{2}\|u-y\|^2\\ f(u) + \frac{1}{2}\|x-u\|^2 - f(y) - \frac{1}{2}\|y-x\|^2 \geq \frac{1}{2}\|u-y\|^2\\ \implies f(y) - f(u) +\frac{1}{2}\|u-y\|^2 + \frac{1}{2}\|y-x\|^2\leq \frac{1}{2}\|x-u\|^2$$

1

There are 1 best solutions below

0
On BEST ANSWER

Based on the comment of gerw, here is the final result. Let $\lambda > 0$, assume that $f(\cdot)-\frac{m}{2}\|\cdot\|^2$ is convex, and let $y=prox_{\lambda f}(x)$, then for all $u$ it holds

$$f(y) - f(u) +\frac{m + \lambda^{-1}}{2}\|u-y\|^2 + \frac{\lambda^{-1}}{2}\|y-x\|^2\leq \frac{\lambda^{-1}}{2}\|u-x\|^2.$$

To see it, denote $g(u) := \lambda f(u) + \frac{1}{2}\|u-x\|^2$ and notice by definition of the prox that

$$0\in \partial g(y).$$

We have that $g(u)$ is $(\lambda m+1)$-strongly convex w.r.t. $\frac{1}{2}\|u-x\|^2$. By this strong convexity of $g(u)$ we have

$$g(u) - g(y) \geq \frac{\lambda m + 1}{2}\|u-y\|^2$$

where we have used the fact that $0\in\partial g(y)$ to eliminate a linear term. Substituting the definition of $g$ we get, for any u,

$$g(u) - g(y) \geq \frac{\lambda m + 1}{2}\|u-y\|^2\\ \lambda f(u) + \frac{1}{2}\|u-x\|^2 - \lambda f(y) - \frac{1}{2}\|y-x\|^2 \geq \frac{\lambda m + 1}{2}\|u-y\|^2\\ \implies f(y) - f(u) +\frac{m + \lambda^{-1}}{2}\|u-y\|^2 + \frac{\lambda^{-1}}{2}\|y-x\|^2\leq \frac{\lambda^{-1}}{2}\|u-x\|^2.$$

And so there is an additional $\frac{m}{2}\|u-y\|^2$ term on the left hand side that pops up when $f$ is $m$-strongly convex with respect to $\frac{1}{2}\|\cdot\|^2$.