Finding the squared expectation of a random variable that is a composite of a summation of random variables.

Question

Given the following, what is the squared expectation of random variable g(x)?

g(x) = $\pi$f($x_1$) + (1-$\pi$)f($x_2$), and we are told simply that $f_1$ and $f_2$ have means $\mu_1$ and $\mu_2$ and variances $\sigma_1^2$ and $\sigma_2^2$ respectively. Furthermore, pi is "some number" existing between 0 and 1, inclusive.

I have been working on this problem, and my current solution is:

E[$X^2$] = $\pi^2$$\sigma_1^2$ + $(1 - \pi)^2\sigma_2^2$ + $\sigma_1\sigma_2$ + $\pi\mu_1$ + $(1 - \pi)\mu_2$

My reasoning is that

1. E[g(x)] = $\pi\mu_1 + (1-\pi)\mu_2$, since pi is a constant, and according to the rules of expectations, the constant should be simply multiplied against the expectation of the density it is a coefficient to.

2. $E[X^2] = E[g^2(x)] = Var(X) + E[X]$, which is a rearrangement of the definition of the variance.

3. Said variance ought to be the first three terms of my solution, if I understand it right, and the constants are squared according to the rule that $Var(aX) = a^2Var(X)$. The third term comes from the fact that $Var(X+Y) = Var(X) + Var(Y) + 2Cov(X,Y)$, and that the $Cov(X,Y) <= SD(X)SD(Y)$

4. Put everything together, and we should arrive at my current solution.

That being said, the question is a multiple choice one, and none of the possible solutions match mine. I'm not confident that this solution I have can be simplified to arrive at one of the choices either.

Answer 1

You seem to be confusing the weighted sum of random variables $$Y = \pi X_1 + (1-\pi)X_2 \tag{1}$$ and a mixture density of random variables $$f_Y(y) = \pi f_{X_1}(y) + (1-\pi) f_{X_2}(y). \tag{2}$$

These are not the same thing. The first one is a weighted sum of the outcomes; the second one is a weighted sum of the probability density functions. Judging from your notation, you seem to be referring to the latter, but your method of solution seems to imply the former.

For the sake of clarity, suppose $X_1$ and $X_2$ are independent random variables, and have means $\mu_1$, $\mu_2$ and variances $\sigma_1^2$, $\sigma_2^2$, respectively. Let $f_{X_1}$ and $f_{X_2}$ be their probability densities. Then by linearity of expectation, the weighted sum in $(1)$ obeys $$\operatorname{E}[Y] = \pi \operatorname{E}[X_1] + (1-\pi) \operatorname{E}[X_2] = \pi \mu_1 + (1-\pi) \mu_2, \tag{3}$$ $$\operatorname{Var}[Y] \overset{\text{ind}}{=} \pi^2 \operatorname{Var}[X_1] + (1-\pi)^2 \operatorname{Var}[X_2] = \pi^2 \sigma_1^2 + (1-\pi)^2 \sigma_2^2. \tag{4}$$ Note the variance calculation is linear because $X_1$ and $X_2$ are independent. From these, the second moment is $$\begin{align} \operatorname{E}[Y^2] &= \operatorname{Var}[Y] + \operatorname{E}[Y]^2 \\ &= \pi^2 \sigma_1^2 + (1-\pi)^2 \sigma_2^2 + (\pi \mu_1 + (1-\pi) \mu_2)^2 \\ &= \pi^2 (\mu_1^2 + \sigma_1^2) + (1-\pi)^2 (\mu_2^2 + \sigma_2^2) + 2\pi(1-\pi)\mu_1 \mu_2. \tag{5} \end{align}$$

Now let us consider the mixture density case. We have

$$\begin{align} \operatorname{E}[Y] &= \int_{y = -\infty}^\infty y \left(\pi f_{X_1}(y) + (1-\pi) f_{X_2}(y)\right) \, dy \\ &= \pi \int_{y=-\infty}^\infty y f_{X_1}(y) \, dy + (1-\pi) \int_{y=-\infty}^\infty y f_{X_2}(y) \, dy \\ &= \pi \operatorname{E}[X_1] + (1-\pi) \operatorname{E}[X_2] \\ &= \pi \mu_1 + (1-\pi) \mu_2. \tag{6} \end{align}$$

So the mean of the mixture distribution is the same as the weighted sum. But what about the second moment? You should already be able to see how it will turn out:

$$\begin{align} \operatorname{E}[Y^2] &= \int_{y = -\infty}^\infty y^2 \left( \pi f_{X_1}(y) + (1-\pi) f_{X_2}(y) \right) \, dy \\ &= \pi \int_{y=-\infty}^\infty y^2 f_{X_1}(y) \, dy + (1-\pi) \int_{y=-\infty}^\infty y^2 f_{X_2}(y) \, dy \\ &= \pi \operatorname{E}[X_1^2] + (1-\pi) \operatorname{E}[X_2^2] \\ &= \pi (\mu_1^2 + \sigma_1^2) + (1-\pi) (\mu_2^2 + \sigma_2^2). \tag{7} \end{align}$$

Now, you might object: why would we not have $$\operatorname{E}[Y^2] \overset{?}{=} \int_{y=-\infty}^\infty y^2 \left( \pi f_{X_1}(y) + (1-\pi) f_{X_2}(y)\right)^2 \, dy?$$ Remember, $Y$ has density according to $(2)$, so the moments of $Y$ are given by $$\operatorname{E}[Y^k] = \int_{y=-\infty}^\infty y^k f_Y(y) \, dy.$$ You don't square the density. So now it is plainly obvious that $(1)$ and $(2)$ do not mean the same thing.

As an exercise for the reader, compute the variance of the mixture distribution; i.e., $\operatorname{Var}[Y]$ when $(2)$ is true (and again, independence of $X_1$ and $X_2$ holds).

Answer 2

Suppose $X$ has a mixture density of the form $$g(x)=\pi f_1(x)+(1-\pi)f_2(x)\,,$$ where $f_i$ is the density of some $X_i$.

In other words,

$$X=\begin{cases} X_1 &, \text{ with probability }\pi \\ X_2 &, \text{ with probability } 1-\pi \end{cases}$$

This corresponds to the weighted sum

$$X=BX_1+(1-B)X_2\,,$$ where $B$ is a $\text{Bernoulli}(\pi)$ variable independent of $(X_1,X_2)$.

Therefore,

\begin{align} \operatorname E(X^2)&=\operatorname E(B^2)\operatorname E(X_1^2)+\operatorname E[(1-B)^2]\operatorname E(X_2^2)+2\operatorname E[B(1-B)]\operatorname E[X_1X_2] \\&=\pi(\sigma_1^2+\mu_1^2)+(1-\pi)(\sigma_2^2+\mu_2^2) \end{align}

The product term vanishes simply because $B^2=B$ with probability $1$.