Combining multiple variances weighted by probability

Question

How do you calculate the overall variance of a situation where there are multiple possible events with known probabilities, and each event has a different and known variance (and expected value)?

As an example, imagine there are three possible carnival games you can play. Each has its own expected value of $E_1$, $E_2$, and $E_3$. Each has its own variance of $V_1$, $V_2$, $V_3$. You roll a fair die to determine which game you're going to play. If the roll is a 1, you play game 1. If the roll is a 2 or 3, you play game 2. If the roll is a 4, 5, or 6, you play game 3.

What is the variance of this situation?

I would think you just do a weighted average of the three variances. However, that doesn't make sense for the trivial case where all three games have a variance of zero (they always cost/pay a fixed amount) because the weighted average of the variances would be zero, but clearly the overall situation has a non-zero variance with the different expected values for the different games.

Answer 1

I would think you just do a weighted average of the three variances. However, that doesn't make sense for the trivial case where all three games have a variance of zero (they always cost/pay a fixed amount) because the weighted average of the variances would be zero, but clearly the overall situation has a non-zero variance with the different expected values for the different games.

Indeed.   Good work in thinking twice about this.   It is just a little more involved.   There's is an additional component to the total variance.

---

Let us take $G$ as number for the game, and $R$ as the result for the game.   We are given these measures:

$$\mathsf E(R\mid G)=E_1{\cdot}\mathbf 1_{G=1}+E_2{\cdot}\mathbf 1_{G=2}+E_3{\cdot}\mathbf 1_{G=3}\\\mathsf{Var}(R\mid G)=V_1{\cdot}\mathbf 1_{G=1}+V_2{\cdot}\mathbf 1_{G=2}+V_3{\cdot}\mathbf 1_{G=3}\\\mathsf P(G{=}g)=\tfrac 16{\cdot}\mathbf 1_{g=1}+\tfrac 13{\cdot}\mathbf 1_{g=2}+\tfrac 12{\cdot}\mathbf 1_{g=3}$$

---

So you should know how to find the Expectation of the Result by the Law of Total Expectation:$$\begin{align}\mathsf E(R)&=\mathsf E(\mathsf E(R\mid G))\\[1ex]&=\sum_g \mathsf P(G{=}g)\cdot\mathsf E(R\mid G{=}g)\\[1ex]&=\tfrac 16 E_1+\tfrac 13 E_2+\tfrac 12E_3\end{align}$$

---

What you want is to find the Variance of the Result, and to do so you likewise use the Law of Total Variance

$$\mathsf {Var}(R)=\mathsf E(\mathsf{Var}(R\mid G))+\mathsf {Var}(\mathsf E(R\mid G))$$

Now, just apply the definitions for expectation and variance.

$$\begin{align}\mathsf E(\mathsf{Var}(R\mid G))&=\sum_g \mathsf P(G{=}g)\cdot\mathsf {Var}(R\mid G{=}g)\\[1ex]&= \phantom{\tfrac 16V_1+\tfrac 13V_2+\tfrac 12V_3}\\[3ex]\mathsf {Var}(\mathsf E(R\mid G))&=\sum_g \mathsf P(G{=}g)\cdot\mathsf E(R\mid G{=}g)^2-\mathsf E(\mathsf E(R\mid G))^2\\[1ex]&=\phantom{\tfrac 16E_1^2+\tfrac 13E_2^2+\tfrac 12E_3^2-\big(\tfrac 16E_1+\tfrac 13E_2+\tfrac 12E_3\big)^2}\end{align}$$

Answer 2

You need to compute the weighted average $\mu_1$ and $\mu_2$ of the first two moments $\mu_{1,k}$ and $\mu_{2.k}=V_k+\mu_{1,k}^2$ and then compute the variance $V=\mu_2-\mu_1^2$

$\mu_1=\frac{\sum_{k=1}^3 k\mu_{1,k}}{6}$ $\mu_2=\frac{\sum_{k=1}^3 k\mu_{2,k}}{6}$

Note that the example your questioned. $V=\frac{\sum_{k=1}^3 (k\mu_{1,k})^2}{6}-(\frac{\sum_{k=1}^3 k\mu_{1,k}}{6})^2\ne 0$

Answer 3

The situation you stated can be described by the following setup:

we have three random variables $X_1, X_2,X_3$ representing the three games with respective expected value and variance. Additionally we define a fourth random variable, say $N$, which is independent (I assume rolling the die is independent of the other games) of the other three and represents the die you roll. We are interested in $Var(X_N)$.

Now let's start with the expected value of $X_N$:

$$ \mathbb{E}[X_N] = \sum_{k=1}^{3} \mathbb{E}[X_N \mathbb{1}\{N=k\}] =\sum_{k=1}^{3} \mathbb{P}(N=k) \mathbb{E}[X_k] $$ where we used the independence of $N$. Similarly we get for the second moment $$ \mathbb{E}[X_N^2] =\sum_{k=1}^{3} \mathbb{P}(N=k) \mathbb{E}[X_k^2]. $$ So in both cases the idea of using weighted sums works out, but remember that we use the squared expectation in the formula for the variance. Therefore, in total the above yields $$ Var(X_N) = \sum_{k=1}^{3} \mathbb{P}(N=k) \mathbb{E}[X_k^2] - \left( \sum_{k=1}^{3} \mathbb{P}(N=k) \mathbb{E}[X_k] \right)^2. $$ Now plug in the probability mass function of $N$ i.e. the die and the respective values for first and second moments of your game.