Proof on property of convex and $\beta$-smooth function - missing term?

1.3k Views Asked by At

I am using this set of notes. On page 268 it writes:

enter image description here

where \eqref{1} in the lemma is

In particular this lemma shows that if $f$ is convex and $\beta$-smooth, then for any $x,y \in \Bbb{R}^n$, one has $$0 \le f(x) - f(y) - \nabla f(y)^\top (x - y) \le \frac\beta2 \lVert x - y \rVert^2. \tag{3.4}\label1 $$

My question is simple: are we missing a $\dfrac{\beta}{2}\|x - z\|^2$ term in the inequality in the encircled?

If not, is it because a function $f$ that satisfies \eqref{1} implies it is convex and $\beta$-smooth (i.e., $\Leftarrow$, note: this was not proved)?

1

There are 1 best solutions below

3
On BEST ANSWER

No, the book's proof is alright.

Explanation of the book's proof

Equation $(3.4)$ actually consists of two inequalities. $\require{action}$ \begin{align} f(x) - f(y) &\ge \mathtip{\nabla f(y)^\top (x-y)}{f \text{ is convex}} \tag1 \label{cvx}\\ f(x) - f(y) - \nabla f(y)^\top (x - y) &\le \mathtip{\frac\beta2 \lVert x - y \rVert^2}{\mbox{Lemma 3.4}} \tag2\label{lem34} \end{align}

From the term $\dfrac{\beta}{2}\|z - y\|^2$ is the proof, it is obvious that \eqref{lem34} is applied to $f(z) - f(y)$, so that $$ \mathtip{ f(z) - f(y) - \nabla f(y)^\top (z - y) \le \frac\beta2 \lVert z - y \rVert^2 }{x \leftarrow z, y \leftarrow y \text{ in } \eqref{lem34}} \tag3\label{lem35} $$ This gives $$ f(z) - f(y) \le \nabla f(y)^\top (z - y) + \frac\beta2 \lVert z - y \rVert^2 \tag4\label{lem35b} $$ It remains to show $$ f(x) - f(z) \le \nabla f(x)^\top (x - z). \tag5\label{lem35c}$$ It seems contradictory to \eqref{cvx}, but since \eqref{cvx} holds for any $x,y \in \Bbb{R}^n$, by multiplying $-1$ on both sides, \eqref{lem35c} is equivalent to $$ \mathtip{f(z) - f(x) \ge \nabla f(x)^\top (z - x). \tag6\label{lem35d}} {x \leftarrow z, y \leftarrow x \text{ in } \eqref{cvx}} $$ Add up \eqref{lem35b} and \eqref{lem35c} to give \begin{align} &\phantom{x=} f(x) - f(y) \\ &= f(x) - f(z) + f(z) - f(y) \\ &\le \mathtip{\nabla f(x)^\top (x - z)}{\text{inequality } \eqref{lem35c}} + \mathtip{\nabla f(y)^\top (z - y) + \frac\beta2 \lVert z - y \rVert^2}{\text{inequality } \eqref{lem35b}} \tag*{$\square$} \end{align}

Why the claimed missing term should not be used?

Since OP talks about $\dfrac{\beta}{2}\|x - z\|^2$, let me show you how we can mess up thing when we apply \eqref{lem34} on $f(x) - f(z)$ to get this term. $$ f(x) - f(z) - \nabla f(z)^\top (x - z) \le \frac\beta2 \lVert x - z \rVert^2 \tag7\label{wrong} $$ Since $z:= y - \dfrac1\beta (\nabla f(y) - \nabla f(x))$, $\nabla f(z)$ in \eqref{wrong} will become too complicated to be manipulated to give a meaningful result.