Approximation of Proximal Operator When $ \lambda $ Is Small

Question

I read that for small $\lambda$, the proximal operator will satisfy

$$\mathbf{prox}_{\lambda f}(x)=x-\lambda\nabla f(x)+o(\lambda)$$

How to prove it?

Here

$$\mathbf{prox}_{\lambda f}(x) = \arg \min\limits_y f(y)+\frac{1}{2\lambda}\|x-y\|^2_2$$

Answer 1

Here is one possible approach using some strong assumptions:

Suppose $f$ is convex and $C^2$ and $\lambda \ge 0$. Then the problem $P_\lambda: \ \min_y \lambda f(y) + {1 \over 2} \|x-y\|^2 $ has a unique minimum $y$ that satisfies $\lambda \nabla f(y) - {1 \over \lambda} (x-y) = 0$.

Consider the implicit system $\phi(\lambda, y) = \lambda \nabla f(y) - {1 \over \lambda} (x-y) $, and note that $\phi(0,x) = 0$ and ${\partial \phi(\lambda, y) \over \partial y} = \lambda {\partial^2 f( y) \over \partial y^2}+ I > 0$. Hence the implicit function theorem shows that there is some locally defined $\zeta$ such that $\zeta(0) = x$ and $\phi(\lambda, \zeta(\lambda)) = 0$. Furthermore, ${\partial \zeta(0) \over \partial \lambda } = -\nabla f(x)$.

Since $\operatorname{prox}_{\lambda f}(x) = \zeta(\lambda)$ for small $\lambda \ge 0$, we have the desired result.