Approximating one random variable by a function of other random variables

Question

Suppose $(X,Y)$ is a continuous bivariate random variable and we want to approximate $Y$ by a function of $X$ ,that means we seek a function say $h(x)$ whose outcomes are the minimum expected squared distance from the outcome of $Y$ , in other words we want to minimize $$\mathbb{E}[(Y-h(X))^2] = \int_{-\infty}^{\infty}\int_{-\infty}^{\infty}(y-h(x))^2 f(x,y)dxdy$$ My question is ,how can we mathematically prove that the optimal choice of approximating function is the regression function i.e $h(x)= E(Y|x)$ ?

Answer 1

$$ \begin{align*} \mathbb{E}[(Y - h(X))^2] & = \mathbb{E}[\mathbb{E}[Y^2 - 2Yh(X) + h(X)^2|X]] \\ & = \mathbb{E}[Y^2] + \mathbb{E}[h(X)^2- 2\mathbb{E}[Y|X]h(X)] \\ & = \mathbb{E}[Y^2] + \mathbb{E}[h(X)^2- 2\mathbb{E}[Y|X]h(X) + \mathbb{E}[Y|X]^2] - \mathbb{E}[\mathbb{E}[Y|X]^2] \\ & = \mathbb{E}[Y^2] + \mathbb{E}[(h(X) - \mathbb{E}[Y|X])^2] - \mathbb{E}[\mathbb{E}[Y|X]^2] \end{align*}$$

So to minimize the square term in the middle $\mathbb{E}[(h(X) - \mathbb{E}[Y|X])^2]$, we choose $h(X) = \mathbb{E}[Y|X]$. It has a projection interpretation inside.