This is a standard problem. Suppose we are given
$$S = \sum\limits_{i = 1}^N X_i$$ where $X_i, i = 1, \ldots$ are iid random variables, with mean $m$, variance $\sigma^2$ and $N$ is an integer-valued random variable indepedent from all the $X_i$s
Compute $\text{var}(S)$
I have made a mistake somewhere in my solution, can anyone check how this might have happened?
Since $\text{var}(S) = E(S^2) - (E(S))^2$
Computing $E(S)$ by conditioning on $N$, which yields:
$$E(S) = E(E(S|N)) = \sum\limits_{n} E(\sum\limits_{i = 1}^n X_i |N = n) \Pr(N=n) = E(X) \sum\limits_{n} n \Pr(N = n) = E(X)E(N)$$
Similarly,
$$E(S^2) = E(E(S^2|N)) = \sum\limits_{n} E(\sum\limits_{i = 1}^n X_i \sum\limits_{k = 1}^n X_k|N = n) \Pr(N=n) = \sum\limits_{n} E(\sum\limits_{i = 1}^n X_i^2 + \sum\limits_{i = 1}^n\sum\limits_{k = 1, i \neq k}^n X_iX_k|N = n) \Pr(N = n) = \sum\limits_{n} E(\sum\limits_{i = 1}^n X_i^2|N = n) \Pr(N = n) = \sum\limits_{n}n E(X^2) \Pr(N = n) = E(X^2) E(N)$$
Then, $$\text{var}(S) = E(X^2) E(N) - (E(X)E(N))^2 = (\text{var}(X) + E(X)^2)E(N) - (E(x)E(N))^2 = \text{var}(X)E(N) - E(X)^2(E(N) - E(N)^2)$$
But this answer is wrong, the correct solution is:
$$\text{var}(S) = \text{var}(X)E(N) + E(X)^2(E(N^2) - E(N)^2)$$
(note: corrected)
Can anyone point out where I made a mistake?
Your calculations look good! I see an error in computing the expectation of $S^2$: why did you get rid of the cross terms in the expectation? Since the $X_i$ are iid, the cross term is
\begin{align*} \sum_n \sum_{1 \leq i \leq n} \sum_{k \neq i} \mathbb{E}X_i \mathbb{E}X_k \mathbb{P}(N = n) & = (\mathbb{E}X)^2\sum_n \mathbb{P}(N = n) (n^2-n) \\ &= (\mathbb{E}X)^2 \Big(\mathbb{E}(N^2) - \mathbb{E}(N)\Big) \end{align*}
So you should have had
\begin{equation} \mathbb{E}S^2 = \mathbb{E}X^2 \mathbb{E}N + (\mathbb{E}X)^2 \Big(\mathbb{E}N^2 - \mathbb{E}N \Big) = Var(X) \mathbb{E}N + (\mathbb{E}X)^2 \mathbb{E}N^2. \end{equation}
This gives me
$Var(S) = Var(X) \mathbb{E}N + (\mathbb{E}X)^2 \Big(\mathbb{E}N^2 - (\mathbb{E}N)^2\Big) = Var(X) \mathbb{E}N + (\mathbb{E}X)^2 Var(N),$
which is a bit different from what you say the correct answer should be. Are you sure there shouldn't be a $+$ instead of $-$ in front of your final parenthetical expression?