Parameter estimation in multivariate Gaussian

Question

X follows $N_p$(A$\theta$,Q). A is known matrix, Q is unknown covariance matrix, $\theta$ is unknown parameter. I need to estimate $\theta$ and show that A$\hat\theta$ has lesser covariance matrix than $\bar X$ which is unbiased estimator of A$\theta$. First i thought that I need to minimize $\sum(X_i-A\theta)^T(X_i-A\theta)$ with respect to $\theta$ but I when tried to differentiate with respect to $\theta$ I got $\sum A^TX_i/\sum A^TA$. I dont feel like this is wright answer. Any help?

Answer 1

First assume $Q$ is known.

You need to minimise $\sum_i (X_i-A\theta)^TQ^{-1}(X_i-A\theta)=\sum_i (X_i-\overline{X})^TQ^{-1}(X_i-\overline{X})+n(\overline{X}-A\theta)^TQ^{-1}(\overline{X}-A\theta)$. So you need to minimise $(\overline{X}-A\theta)^TQ^{-1}(\overline{X}-A\theta)=(Q^{-1/2}\overline{X}-Q^{-1/2}A\theta)^T(Q^{-1/2}\overline{X}-Q^{-1/2}A\theta)$.

Make the transformation $X\mapsto Q^{-1/2}X=Y\sim N_p(Q^{-1/2}A\theta,I). $

So $(\overline{Y}-Q^{-1/2}A\theta)^T(\overline{Y}-Q^{-1/2}A\theta)$ is to be minimised.

There is a Geometric way to do this minimisation. On $\mathbb R^p$ define the inner product $\langle x,y\rangle=x^Ty$. Then for a given matrix $A$, note that in minimising $(x-A\beta)^T(x-A\beta)$ with respect to $\beta$, the minimum is achived at $A\beta=P_A(x)$ where $P_A$ is the orthogonal projection matrix onto $\mathcal (A)$, with the form $P_A=A(A^TA)^{-}A^T$. Any $\beta$ satisfying $A\beta=P_A(x)$ will minimise $(x-A\beta)^T(x-A\beta)$. In particular you may choose $\beta=(A^TA)^{-}A^x$.

So to minimise $(\overline{Y}-Q^{-1/2}A\theta)^T(\overline{Y}-Q^{-1/2}A\theta)$ you want to find $\theta$ such that $Q^{-1/2}A\theta=P_{Q^{-1/2}A}(\overline{Y})=Q^{-1/2}A(A^TQ^{-1}A)^{-}A^TQ^{-1/2}\overline{Y}=Q^{-1/2}A(A^TQ^{-1}A)^{-}A^TQ^{-1/2}Q^{-1/2}\overline{X}$

This implying a solution to $\theta$ is $$\theta_{MLE}=(A^TQ^{-1}A)^{-}A^TQ^{-1}\overline{X}$$.

This is for known $Q$. For unknown $Q$, what you do is you first maximise the likelihood with respect to $Q$, the MLE for $Q$ being $S=\dfrac{1}{n}\sum_{i=1}^n (X_i-\overline{X})(X_i-\overline{X})^T$. Then define the inner product as above with $Q$ replaced by $S$, and everything follows as above, so finally, your $\theta_{MLE}$ is same as before, with $Q$ replaced by $S$.

It is trivial to check that $A\theta_{MLE}$ is unbiased for $A\theta$. To show that it has smaller variance than any other estimator is a consequence of the celebrated Gauss-Markov Theorem.

Honestly, this is an exercise? I could cover a whole chapter of a book writing this! .