Clever Solutions: Find $\min_{a,b} E[(V -(aU+b))^2]$ without differentiation?

Question

Let $(U,V)$ be joint random variables (assume zero means for simplicity). It is well known that \begin{align} \min_{a,b} E[(V -(aU+b))^2], \end{align} is minimized by $a= \frac{E[VU]}{E[U^2]}$ and $b=0$. This answer results in what is know as the best linear estimator.

The classical proof of this expands the expression and finds partial derivatives with respect to $a$ and $b$.

My question: Are there alternative proofs that do not use differentiation? For example, can this be found via some known inequalities like Jensen or Cauchy-Schwarz? In other words, I am looking for interesting and unique solutions to this old problem.

Edit I didn't ask this originality, but can it be also done with minimal expansion.

Answer 1

Expanding, \begin{align*} \mathbb{E}(V-(aU+b))^2 &= \mathbb{E}V^2 - 2 (a \mathbb{E} UV + b \mathbb{E} V) + \mathbb{E}(aU+b)^2 \\ &= (a^2 \mathbb{E}U^2 - 2a \mathbb{E} UV) + b^2 + \mathbb{E}V^2 \end{align*} Note that these are separable quadratics in $a, b$ (since the cross term $ab$ vanishes because we assumed $\mathbb{E} U = 0$), and each is minimized at the vertex, given by $x = -\gamma_1/2\gamma_2$ in $\gamma_2 x^2 + \gamma_1 x + \gamma_0 = 0$. Hence, \begin{align*} a &= \frac{2 \mathbb{E} UV}{2 \mathbb{E}U^2} = \frac{\mathbb{E} UV}{\mathbb{E}U^2} \\ b &= -\frac{0}{2\cdot 1} = 0 \end{align*}

Answer 2

If you consider $E(XY)$ as scalar product of random variables, then your have the following meaning:

• $E(X^2)$ — is the squared length
• $E((X-Y)^2)$ — is the squared distance between $X$ and $Y$

Using this interpretation your problem is to find the random variable $aU+b$ that is the closest one to the random variable $V$. But all random variables of the form $aU+b$ are linear combinations of the random variable $U$ and constant $1$. So you just need to find the projection of random variable $V$ on this plane.

The difference $V - (aU+b)$ should be orthogonal to $U$ and $1$. The corresponding scalar product should be zero. So you have two equations $$ \begin{cases} \langle V - (aU+b), 1 \rangle = E(V - (aU+b))=0 \\ \langle V - (aU+b), U \rangle = E((V - (aU+b))\cdot U)=0 \\ \end{cases} $$

From this system your get $a$ and $b$ :)