Prove $\bar{X}$ and $X_i-\bar{X}$ are independent if $X_i's$ are independently normally distributed

Question

The statement comes from the book statistics and data analysis page 179, and I just wonder how to prove that $\bar{X}$ and $X_i-\bar{X}$ are independent if $\require{enclose}\enclose{horizontalstrike}{Cov(\bar{X},X_i-\bar{X})=0}$ and $X_i's$ are independently normally distributed.

Answer 1

(Note: In this answer I'll only consider standard normal $X_i$s, since it's no harder to do when they have other means and variances)

One way of doing this is with moment generating functions. For a bivariate random variable $Z = (Z_1, Z_2)$, $Z_1$ and $Z_2$ are independent if and only if the moment generating function $M_Z(\mathbf{s}) = \mathbf{E} e^{\mathbf{s}^T Z}$ factors into a product of a functions of $s_1$ and $s_2$ only, where $\mathbf{s} = (s_1, s_2)$.

Now, the thing to notice is that $(\bar{X}, X_i - \bar{X})$ is a linear transformation of the whole sample $X = (X_1, \dotsc, X_n)$. Namely, $$ \begin{pmatrix}\bar{X} \\ X_i - \bar{X}\end{pmatrix} = \begin{pmatrix}\mathbf{1}^T/n \\ e_i^T - \mathbf{1}^T/n\end{pmatrix} X := H X $$ where $\mathbf{1}$ is a vector of all $1$s and $e_i$ is the vector of all $0$s with a $1$ in the $i$th entry.

We can use this to write down the MGF: \begin{align*} \mathbf{E} \exp\biggl\{\mathbf{s}^T \begin{pmatrix}\bar{X} \\ X_i - \bar{X}\end{pmatrix}\biggr\} = \mathbf{E} \exp\{\mathbf{s}^T H X\} = \mathbf{E} \exp\{(H^T\mathbf{s})^T X\} \end{align*}

Then we can use the known moment generating function of a vector of iid normals ($\mathbf{E} e^{\mathbf{t}^T X} = \exp\{\frac{1}{2}\mathbf{t}^T \mathbf{t}\}$) to conclude that $$ \mathbf{E} \exp\{(H^T\mathbf{s})^T X\} = \exp\{\frac{1}{2}(H^T\mathbf{s})^T(H^T\mathbf{s})\} = \exp\{\frac{1}{2}\mathbf{s}^T HH^T \mathbf{s}\}. $$

Now, we can multiply \begin{align*} HH^T &= \begin{pmatrix}\mathbf{1}^T/n \\ e_i^T - \mathbf{1}^T/n\end{pmatrix} \begin{pmatrix}\mathbf{1}/n , & e_i - \mathbf{1}/n\end{pmatrix} \\ &= \begin{pmatrix} \mathbf{1}^T/n \mathbf{1}/n & \mathbf{1}^T/n (e_i - \mathbf{1}/n) \\ (e_i - \mathbf{1}/n)^T \mathbf{1}/n & (e_i - \mathbf{1}/n)^T(e_i - \mathbf{1}/n) \end{pmatrix} \\ &= \begin{pmatrix} 1/n & 0 \\ 0 & 1 - 1/n \end{pmatrix}, \end{align*} so that the MGF becomes $$ \exp\biggl\{\frac{1}{2}\mathbf{s}^T HH^T \mathbf{s}\biggr\} = \exp\biggl\{\frac{1}{2}\biggl(\frac{1}{n}s_1^2 + \frac{n-1}{n}s_2^2\biggr)\biggr\} = e^{\frac{1}{2n}s_1^2} e^{\frac{n-1}{2n} s_2^2}. $$

Since this factors into terms containing only $s_1$ and $s_2$ respectively, we conlcude that $\bar{X}$ and $X_i - \bar{X}$ are independent (and also that they have $N(0, 1/n)$ and $N(0, 1 - 1/n)$ distributions respectively).