Suppose we want to estimate a single mean $\mu_1$ of a bivariate Gaussian, whose covariance matrix is known, but the means $\mu_1$ and $\mu_2$ are unknown.
Let $N$ be the number of joint samples. If one of the means is known and equals zero, i.e., $\mu_2=0$, we can leverage the correlation coefficient through the following estimator
\begin{equation} \overline{\mu}_1 = \frac{\sum X_{i,1}}{N} + \rho \frac{\sigma_1}{\sigma_2}\frac{\sum X_{i,2}}{N} \end{equation}
see, e.g., page 4 of Sampling: regression methods on estimation
In general, \begin{equation} \overline{\mu}_1 = \frac{\sum X_{i,1}}{N} + \rho \frac{\sigma_1}{\sigma_2}\left(\frac{\sum X_{i,2}}{N} -\mu_2\right) \end{equation}
So, if the two means are unknown, why the following idea isn't any better than using the classical sample mean to estimate the unknown mean?
- estimate one of the unknown means, $\mu_2$, e.g., assuming that its variance is low,
- use the information learned about that unknown mean to learn the other unknown mean, $\mu_1$
It seems that the sample mean is the best estimator for the unknown mean. However, it is very counterintuitive to me the fact that
we can leverage $\mu_2$ to learn $\mu_1$ when $\mu_2$ is known
we cannot leverage any partial knowledge about $\mu_2$, e.g., learned while collecting joint samples, and the information from the covariance matrix, to learn more about $\mu_1$ when both $\mu_1$ and $\mu_2$ are unknown.
What am I missing? Is it possible to leverage partial knowledge about $\mu_2$ to estimate $\mu_1$? If so, how? Otherwise, what is the intuition behind the fact that the simple sample mean $ \frac{\sum X_{i,1}}{N}$ is the best that can be done to estimate $\mu_1$ when $\mu_2$ is unknown?
This is a follow-up to:
Estimating two means of bivariate gaussian
See also this sentence:
Of course, if we are interested in estimating only $\mu_1$ the presence of other unknown means cannot make our task any easier.
Why "of course"?
Stein, Charles. Inadmissibility of the usual estimator for the mean of a multivariate normal distribution. Stanford University Stanford United States, 1956.