Conditional Bias Variance Decomposition

80 Views Asked by At

The standard bias variance decomposition says that: $$ E |f(X) - Y|^2 = \int_{\mathbb{R}^d} |f(x) - m(x)|^2 \mu(dx) + E|m(X) - Y|^2, $$ where $\mu$ is some distribution over $X$. I am trying to understand the conditional version of this decomposition: Let $D_n= \{(X_i,Y_i)\}_{i=1}^n$ be an i.i.d. sample and we construct an estimate $m_n(x) := m_n(x,D_n)$ based on the sample. We then have \begin{align*} E [|m_n(X) - Y|^2\mid D_n] &= \int_{\mathbb{R}^d} |m_n(x) - m(x)|^2 \mu(dx) + E|m(X) - Y|^2\\ &= E|m_n(X) - m(X)|^2 + E|m(X) - Y|^2, \end{align*} These are equations 1.1 and 1.2 in the Nonparametric Statistics book by Gyorfi. What I am confused about is the first term in the second equation, should this not be a conditional (on $D_n$) expectation? That is: $$ E [|m_n(X) - Y|^2\mid D_n] = E [|m_n(X) - m(X)|^2 \mid D_n] + E|m(X) - Y|^2 $$

update: I think what is happening is: \begin{align*} E [|m_n(X) - m(X)|^2 \mid D_n] &= \int_{m_n} \int_x |m_n(x) - m(x)|^2 dP(x,m_n|D_n)\\ &= \int_{m_n} \int_x |m_n(x) - m(x)|^2 d\mu(x) dP(m_n|D_n)\\ &= E\left ( \int_x |m_n(x) - m(x)|^2 d\mu(x) \mid D_n \right )\\ &= E\left ( E[|m_n(X) - m(X)|^2]\mid D_n \right )\\ &= E[|m_n(X) - m(X)|^2], \end{align*} where the second equality is due to $X$ being an independent draw, and the last equality is due to the fact that $m_n$ is measurable with respect to $D_n$, so the expectation of the inner expectation conditional on $D_n$ is just equal to the inner expectation almost surely.