A simple proof of McDiarmid's inequality?

252 Views Asked by At

$\newcommand{\Ex}{\mathrm{E}} \newcommand{\Pr}{\mathrm{Pr}} \newcommand{\Ind}{\mathbf{1}} \newcommand{\implies}{\Rightarrow}$ Let $f(x_1, \ldots, x_n)$ be a real valued function of $n$ variables, with each $x_i \in \mathcal{X}$, such that: \begin{equation} \forall (x_1,\ldots,x_{n-1}) \in \mathcal{X}^{n-1}, | f(x_1, \ldots, x_{n-1}, x) - f(x_1, \ldots, x_{n-1}, x') | \leq 1 , \end{equation} where $x, x' \in \mathcal{X}$. Note that McDiarmid's assumes that $f$ is 1-Lipschitz, i.e., the above condition holds if we change any coordinate (and not just the n-th coordinate). The constant 1 is not important, it can be any arbitrary constant.

First, I use the Hoeffding's inequality to show that for any arbitrary $(x_1,\ldots,x_{n-1})$: \begin{equation*} \Pr_{X_n}\{ \hat{\Ex}[f(x_1, \ldots, x_{n-1}, X_n)] - \Ex[f(x_1, \ldots, x_{n-1}, X_n)] \geq t \mid (x_1, \ldots, x_{n-1}) \} \leq e^{-2mt^2}, \end{equation*} where $\hat{\Ex}$ is the average of $f(\cdot)$ over $m$ samples of the $X_n$ while fixing the first $n-1$ variables to $x_1, \ldots, x_{n-1}$.

Next, I take expectation with respect to $X_1, \ldots, X_{n-1}$, of both sides of the equation to get: \begin{gather*} \Pr_{X_n}\{ \hat{\Ex}[f(x_1, \ldots, x_{n-1}, X_n)] - \Ex[f(x_1, \ldots, x_{n-1}, X_n)] \geq t \mid (x_1, \ldots, x_{n-1}) \} \leq e^{-2mt^2} \\ \implies \Ex_{X_n}[ \Ind[\hat{\Ex}[f(x_1, \ldots, x_{n-1}, X_n)] - \Ex[f(x_1, \ldots, x_{n-1}, X_n)] \geq t] \mid (x_1, \ldots, x_{n-1}) ] \leq e^{-2mt^2} \\ \implies \Ex_{X_1, \ldots X_{n-1}}[\Ex_{X_n}\{ \Ind[\hat{\Ex}[f(x_1, \ldots, x_{n-1}, X_n)] - \Ex[f(x_1, \ldots, x_{n-1}, X_n)] \geq t]] \mid (x_1, \ldots, x_{n-1}) ] \leq e^{-2mt^2} \\ \implies \Pr_{X_1, \ldots, X_n}\{ \hat{\Ex}[f(X_1, \ldots, X_n)] - \Ex[f(X_1, \ldots, X_n)] \geq t \} \leq e^{-2mt^2}. \end{gather*} Essentially, I have shown concentration of $f(.)$ with respect to all variables by showing concentration with respect to only one variable and without constructing a Doob Martingale as the standard McDiarmid's proof does. The above is definitely wrong but I am not able to see where.

Also, is the above proof correct if $\mathcal{X}$ is finite or countably infinite?

1

There are 1 best solutions below

0
On

I found the bug in my proof. So I will go ahead and answer my own question. The problem with the above is that if we fix the first $n-1$ variables to $(x_1, \ldots, x_{n-1})$ and then take $m$ independent samples of $X_n$: $X_n^1, \ldots, X_n^m$. Denote $f_i = f(x_1, \ldots, x_{n-1}, X_n^i)$. Then the $f_i$'s are not necessarily independent. Hoeffding's applies only to sum of independent random variables. The proof of McDiarmid's inequality skirts over this issue by constructing a Martingale difference sequence.