Conditional Expectation with transformation

Question

Let $Y_1, Y_2,\ldots, Y_n$ be independent $N(0,1)$ random variables. Define $ X_i =\sum_{j=1}^n c_{i,j}Y_j$, $i=1,2,\ldots,n$, where $c_{i,j}$ are real constants. Show that $E(X_i\mid X_k)=\dfrac{\sum_{j=1}^n c_{i,j}c_{k,j}}{\sum_{j=1}^nc_{k,j}^2}X_k$. What is $\operatorname{Var}(X_i\mid X_k)$?

I can only think of $X$ as a transformation of $Y$, which gives me the result that $X$ is Gaussian with $N(0, CC')$. But I got no idea how to get the conditional expectation and variance.

Answer 1

Conditional distributions in normal families are always linear (which explains partly the ubiquity of the Gaussian distribution in modeling), for example the fact that $$X_i=\alpha_{i,k}X_k+\beta_{i,k}Z_{i,k}+\gamma_{i,k}, $$ for some deterministic coefficients $\alpha_{i,k}$, $\beta_{i,k}$ and $\gamma_{i,k}$, and for a standard normal random variable $Z_{i,k}$ independent of $X_{i,k}$ implies that $$ E(X_i\mid X_k)=\alpha_{i,k}X_k+\gamma_{i,k}, $$ and $$ \mathrm{var}(X_i\mid X_k)=\beta_{i,k}^2. $$ To identify these coefficients, note that, for every $i\ne k$, $$ E(X_i)=E(X_k)=0,\qquad\mathrm{var}(X_i)=\sum_jc_{i,j}^2, $$ and $$ \mathrm{Cov}(X_i,X_k)=\sum_jc_{i,j}c_{k,j}, $$ on the one hand, and that $$ E(X_i)=\gamma_{i,k},\qquad\mathrm{var}(X_i)=\alpha^2_{i,k}\mathrm{var}(X_k)+\beta^2_{i,k}, $$ and $$ \mathrm{Cov}(X_i,X_k)=\alpha_{i,k}\mathrm{var}(X_k), $$ on the other hand. Hence $$ \gamma_{i,k}=0,\qquad\alpha_{i,k}=\frac{\sum\limits_jc_{i,j}c_{k,j}}{\sum\limits_jc_{k,j}^2}, $$ and $$ \beta^2_{i,k}=\sum_jc_{i,j}^2-\frac{\left(\sum\limits_jc_{i,j}c_{k,j}\right)^2}{\sum\limits_jc_{k,j}^2}. $$ Considering the canonical scalar product $\langle\ ,\ \rangle$ and the tuples $c_i=(c_{ik})_k$, this can be rewritten as $$ \alpha_{i,k}=\frac{\langle c_i,c_k\rangle}{\langle c_k,c_k\rangle}, \qquad \beta^2_{i,k}=\langle c_i,c_i\rangle-\frac{\langle c_i,c_k\rangle^2}{\langle c_k,c_k\rangle}. $$

Answer 2

You are right that $X\sim N(0,CC')$. Conditional distributions are somewhat subtler. One can look at $$ \begin{bmatrix} X_i \\ X_j \end{bmatrix} \sim N\left(\begin{bmatrix} 0 \\ 0 \end{bmatrix}, \begin{bmatrix} \sum_{\ell=1}^n c_{i,\ell}^2, & \sum_{\ell=1}^n c_{i,\ell}c_{j,\ell} \\ \sum_{\ell=1}^n c_{i,\ell}c_{j,\ell}, & \sum_{\ell=1}^n c_{j,\ell}^2 \end{bmatrix} \right) = N\left( \begin{bmatrix} 0 \\ 0 \end{bmatrix}, \begin{bmatrix} a & \rho ab \\ \rho ab & b \end{bmatrix} \right), $$ so that $\rho$ is the correlation.

Maybe I'll come back later and write proofs of some of the assertions below, but for now I'll say $$ \begin{align} & E(X_i\mid X_j) = cX_j+d \quad\text{i.e. one needs to fit a straight line,} \\[8pt] & X_i - E(X_i \mid X_j) \text{ is independent of $X_j$, i.e. the $``$errors" are independent of the predictors,} \\[8pt] & \operatorname{var}(X_i \mid X_j) \text{ does not depend on $X_j$ (homoscedasticity)}, \\[8pt] & \mathbb E (\operatorname{var}(X_i\mid X_j)) + \operatorname{var}(\mathbb E(X_i\mid X_j)) = \operatorname{var}(X_i), \\[8pt] & \text{when $X_j$ is $z$ S.D.s above the mean of $X_j$, then $\mathbb E(X_i\mid X_j)$ is $\rho z$ S.D.s above the mean of $X_i$.} \end{align} $$

You can work it out with some algebra based on that. I'll be back later and maybe add something then.