This is an exercise from a functional analysis textbook. I am stuck in it.
Let $H$ be a Hilbert space, $A\in\mathcal{B}(H)$. We call $A$ a contraction operator if $\|A\|\leq 1$ and call $A$ a positive operator if for any $x\in H$, $\langle A(x),x\rangle\geq 0$.
We can prove that for any positive contraction operator $A$ on $H$, $A$ is self-adjoint and the following inequality $$\|x-A(x)\|^2\leq \|x\|^2-\|A(x)\|^2,\quad\forall x\in H$$ holds. (Proof omitted.)
Now give $r$ positive contraction operators $A_1,A_2,\cdots,A_r$ on $H$, let $T=A_1A_2\cdots A_r$, the goal is to prove $$\|x-T(x)\|^2\leq r\left(\|x\|^2-\|T(x)\|^2\right),\quad\forall x\in H.$$
My attempt is to write the LHS of the above inequality as $$\|x-A_1(x)+A_1(x)-A_1A_2(x)+\cdots+A_1A_2\cdots A_{r-1}(x)-A_1A_2\cdots A_r(x)\|^2$$ and expand it, but it doesn’t seem to work.
You were on the right track, you just have to to do it in the opposite order. We have \begin{align*} \|x-Tx\|^2&=\|x-A_rx+A_rx-A_{r-1}A_rx+\dots+A_2\dots A_rx-A_1\dots A_rx\|^2\\ &\leq (\|x-A_rx\|+\|A_rx-A_{r-1}(A_rx)\|+\dots+\|A_2\dots A_rx-A_1(A_2\dots A_r)x\|)^2\\ &\leq r(\|x-A_r x\|^2+\|A_r x-A_{r-1}(A_r x)\|^2+\dots+\|A_2\dots A_rx-A_1(A_2\dots A_r)x\|^2)\\ &\leq r(\|x\|^2-\|A_r x\|^2+\dots+\|A_2\dots A_r x\|^2-\|A_1\dots A_r x\|^2)\\ &=r(\|x\|^2-\|Tx\|^2). \end{align*} The first inequality is simply the triangle inequality, the second one is the Cauchy-Schwarz inequality, and the third the inequality for the individual positive contractions from the question.