Gradient of $\|Ax-P_CAx\|^2$ where $A$ is a bounded linear operator and $P_C$ is a metric projection.

Question

Let $H$ be a Hilbert space, $C$ be a closed convex set in $H$. Metric projection $P_C$ from $H$ onto $C$ is defined as, $P_C(x) = \operatorname{arg min}_{y \in C} \{\|x-y\|\}$. $A$ is a bounded linear operator from $H$ to $H$. Let $g(x)=\|Ax-P_CAx\|^2$, then $\nabla g(x)=\text{?}$

Answer 1

I'll assume you're working in a real Hilbert space. I believe your function is not differentiable in a complex Hilbert space. Take $H = \mathbb{C}$, $C = \mathbb{R}$, $A = I$ (the identity), $P_C$ the orthogonal projection onto $C$. Then for $z = x + iy$, $Az = z$ and $P_CAz = x = Re(z)$, and so $g(z) = Im(z)^2$, which is not differentiable. Correct me if this is an invalid counterexample.

Under this assumption, expand $g$ to get

\begin{align} g(x) &= (Ax,Ax)_{H}+(P_CAx,P_CAx)_H-2(P_CAx,Ax)_H \\ &= (x,A^{*}Ax)_H+(x,B^*Bx)_H-2(x,B^*Ax)_H \\ &= (x,Mx)_H \end{align} where $B = P_CA$ and $M = A^*A + B^*B - 2 B^*A$. Now note

\begin{align} g(x+h) - g(x) &= (x,Mh)_H+(h,Mx)_H+(h,Mh)_H \\ &= (h,(M^*+M)x)_H+(h,Mh)_H \Rightarrow \\ g(x+h) - g(x) - R(x)h &= (h,Mh)_H \end{align} where $R: H \to B(H,\mathbb{R})$, $R(x)h = (h,(M^*+M)x)_H$ is well-defined because $P_C,A$ bounded implies $M$ is bounded. Now by the Cauchy-Schwarz inequality,

\begin{align} \frac{\|g(x+h)-g(x)-R(x)h\|_H}{\|h\|_H} \leq \|Mh\|_H \leq \|M\|_{B(H,H)}\|h\|_H \end{align} which goes to $0$ as $\|h\|_H \to 0$. So $\nabla g = R.$ In other words, the $\nabla g$ associates to each $x \in H$ the Riesz map associated to the element $(M^*+M)x$ where $M$ is defined as before. Since $P_C^2 = P_C$ and $P_C$ is self-adjoint, $(M^* + M)x = 2A^*(I-P_C)Ax = 2A^*(Ax-P_CAx)$.

Answer 2

The derivative of $f \colon x\mapsto \|x-P_Cx\|^2$ is equal to $2(x-P_Cx)$.
Your function $g$ is the composition $f\circ A$.
Hence, by the chainrule,

$$\nabla g(x) = A^*(\nabla f(Ax)) = 2A^*(Ax-P_C(Ax)).$$