I am finishing my statistics course at school and this week, we learned how to estimate a random variable X to minimize MSE.
One of the cases is to estimate X given another random variable Y. I was told that the MMSE estimator for X given Y is
$$\hat{X}_M(Y)= E[X|Y]$$
However, the MMSE estimator, most of the time, is hard to compute. Instead, it is easier to compute the LMSE estimator for X given Y:
$$\hat{X}_L(Y)= aY + b$$
Here is my question. What if X and Y are independent random variables? If they are independent, then $f_{X|Y}(x|y) = f_X(x)$. Therefore,
$$E[X|Y] = \int_{-\infty}^{\infty} xf_{X|Y}(x|y)dx = \int_{-\infty}^{\infty} xf_{X}(x)dx = E[X]$$ (in case X is continuous)
So to estimate X given Y when X and Y are independent is the same as to do a blind estimation of X. Am I correct? What about the LMSE estimator?
Thank you all!
As lafinur noted in the comments, you are correct. $X$ and $Y$ being independent means that they don’t contain any information about each other, so knowing $Y$ doesn’t change how you can estimate $X$.
To show this concretely for the LMSE , minimize the mean square error:
\begin{eqnarray*} \def\E{\mathsf E} \frac{\partial}{\partial a}\E\left[\left(\hat X_L(Y)-X\right)^2\right] &=& \frac{\partial}{\partial a}\E\left[\left(aY+b-X\right)^2\right] \\ &=& 2E\left[Y(aY+b-X)\right]\\ &=& 2\left(a\E\left[Y^2\right]+b\E[Y]-\E[XY]\right)\\ &=& 2\left(a\E\left[Y^2\right]+b\E[Y]-\E[X]\E[Y]\right)\\ &\stackrel!=&0 \end{eqnarray*}
and
\begin{eqnarray*} \frac{\partial}{\partial b}\E\left[\left(\hat X_L(Y)-X\right)^2\right] &=& \frac{\partial}{\partial b}\E\left[\left(aY+b-X\right)^2\right] \\ &=& 2\E\left[aY+b-X\right]\\ &=& 2\left(a\E\left[Y^2\right]+b\E[Y]-\E[XY]\right)\\ &=& 2\left(a\E\left[Y\right]+b-\E[X]\right)\\ &\stackrel!=&0\;. \end{eqnarray*}
Subtract $\E[Y]$ times the second equation from the first to show that $a=0$ if $\mathsf{Var}[Y]=\E\left[Y^2\right]-\E[Y]^2\ne0$, and then solve the second equation to obtain $b=\E[X]$. Thus, in this case the LMSE is just $\hat X_L(Y)=\E[X]$.
If $\mathsf{Var}[Y]=0$ (i.e. if $Y$ is almost surely constant), the two equations are linearly dependent, and any estimator of the form $\hat X_L(Y)=aY+\E[X]-a\E[Y]$ for any $a$ will do (since $Y$ is almost surely equal to $\E[Y]$).