MGF bound for non-centered random variables?

292 Views Asked by At

So I was trying to prove the following fact: if $|X| \leq 1$ is a random variable with mean $\mu$, then $\mathbb{E} e^{t (X - \mu)} \leq e^{t^2/2}$, for any real $t$.

I can show this in the special case $\mu = 0$: for $x \in [-1, 1]$ we note by convexity $$ e^{tx} \leq \frac{1-x}{2} e^{-t} + \frac{1 + x}{2} e^{t}. $$ Hence, taking expectations: $\mathbb{E} e^{t (X -\mu)} \leq \cosh(t) \leq e^{t^2/2}$.

Any ideas how to extend this argument to general $\mu$?

1

There are 1 best solutions below

1
On

It suffices to show that for a random variable $Y \in [a,b]$ with $\mathbb E Y =0$, we have that $\mathbb E e^{tY} \leq e^{t^2(b-a)^2/8}$. Why? Let $Y = X - \mu$, then $\mathbb E Y = 0$, $b = 1 - \mu$, $a = -1 - \mu$, $(b-a)^2 = 4$, and the bound becomes $e^{t^2/2}$.

We use the same argument as you outlined. Fix a $t$. Using the convexity of $e^{ty}$, for any $y \in [a,b]$: \begin{aligned} e^{ty} \leq \frac{b-y}{b-a} e^{ta} + \frac{y-a}{b-a} e^{tb}. \end{aligned} Therefore, using the linearity of expectation and $\mathbb E Y = 0$, we get \begin{aligned} \mathbb E e^{tY} \leq \frac{be^{ta} - ae^{tb}}{b-a} =: \varphi(t). \end{aligned} Make the following substitution: Let $\phi(t):= \log \varphi(\frac{t}{b-a})$ and $p = -a/(b-a)$. We need to show that $\phi(t) \leq t^2/8$.
We have that $\phi(t) = -tp + \log(1 - p + pe^{t})$.

Notice that $\phi(0) = 0$, $\phi'(0) = 0$, and \begin{aligned}\phi''(t) = \frac{(1-p)(pe^t)}{(1-p+pe^t)^2} \leq \frac{1}{4} ,\end{aligned} where the last inequality holds for all $t$ by the AM-GM inequality. Applying Taylor's theorem, we get that $\phi(t) \leq t^2/8$.

Remark: This is a very famous result called Hoeffding's Lemma. I followed the proof given on Wikipedia as it was the closet to your original approach. For alternate proofs, see, for example, Lemma 2.2 in the book Concentration Inequalities: A Nonasymptotic Theory of Independence .