Linear regression: Distribution of sum of squares from moment generating function

Question

To find the pdf of Z, how do we combine E and H? And hence its MGF? Thanks

Answer 1

First establish the following properties of $H$:

(a) $H^T=H$, so $H$ is symmetric.

(b) $H^2 = H$, so $H$ is idempotent. It follows that the eigenvalues $\lambda_1,\cdots,\lambda_n$ of $H$ are either $0$ or $1$, and the number of unit eigenvalues is $\sum\lambda_i$ which equals the trace of $H$.

(c) The trace of $H$ is $n -\operatorname{tr}[x(x^Tx)^{-1}x^T]=n-p$.

For (d), check that $\hat E^T\hat E=E^THE$ so the desired MGF is $$M_Z(s):={\mathbb E} \exp\big((s/\sigma^2)\hat E^T\hat E\big)={\mathbb E}\exp( s U^THU) =\int_{u\in{\mathbb R^n}} \exp(su^THu)f_U(u)\,du\tag1$$ where $U\sim N(0, I)$. Plug in the joint density of $U$: $$ \int\exp(su^THu)\frac1{(2\pi)^{n/2}}\exp( -{\textstyle\frac12}u^Tu)\,du=\frac1{(2\pi)^{n/2}}\int\exp\big(-{\textstyle\frac12}u^T(I-2sH)u\big)\,du\tag2 $$ This has the desired form, with $M:=(I-2sH)^{-1}$. Recalling the density of the multivariate normal, we know $\int \exp(-{\textstyle\frac12}u^TM^{-1}u)\,du=(2\pi)^{n/2}(\det M)^{1/2}$. Therefore $$M_Z(s) = \left(\det(I-2sH)^{-1}\right)^{1/2}=\frac1{\big(\det(I-2sH)\big)^{1/2}}\tag3$$

Now the eigenvalues of $I-2sH$ are $1-2s$ (with multiplicity $n-p$) and $1$ (multiplicity $p$) so $\det(I-2sH)=(1-2s)^{n-p}$.

For (e), we recognize the MGF of $Z$ is that of a chi-squared distribution with $n-p$ degrees of freedom, which has expectation $n-p$. So an unbiased estimator of $\sigma^2$ is $(\hat E^T\hat E)/(n-p)$.