What is the intuition behind the Bregman divergence of three points?

1.7k Views Asked by At

Let $\phi:\Omega\rightarrow\mathbb{R}$ ($\Omega \subset\mathbb{R}^n$) be a continuously-differentialble real-valued and strictly convex function defined on a closed convex set $\Omega$. For each two points the Bregman divergence is defined as:

$$D_\phi(x,y)=\phi(x)-\phi(y)-<\nabla \phi(y),x-y>$$

If we write the Bregman divergence for $y$ and $z$

$$D_\phi(y,z)=\phi(y)-\phi(z)-<\nabla \phi(z),y-z>$$

We have the following identity

$$D_\phi(x,y)+D_\phi(y,z)=D_\phi(x,z)+<\nabla \phi(z)-\nabla \phi(y),x-y>$$

Is there any geometrical interpretation to explain two sides of this equality based on the definition of the Bregman divergence and the projection of $\nabla \phi(z)-\nabla \phi(y)$ on $x-y$ or $x-y$ on $\nabla \phi(z)-\nabla \phi(y)$?

To have better understanding one can use the following picture.

enter image description here

1

There are 1 best solutions below

0
On BEST ANSWER

It's a generalization of the identity (some call it Pythagorean identity) for Hilbertian norms,

$$\frac{1}{2}\|x-z\|^2 = \frac{1}{2}\|x-y+y-z\|^2 = \frac{1}{2}\|x-y\|^2 + \frac{1}{2}\|y-z\|^2 + \langle x-y,y-z\rangle$$