The question is as follows:
Suppose that the conditional distribution of $N$, given that $Y = y$, is Poisson with mean $y$. Further suppose that $Y$ is a gamma random variable with parameters $(r, λ)$, where $r$ is a positive integer.
I want to get Var(N)
I know that $$\text{Var}(N) = E(N^2) - (E(N))^2 = E(\text{Var}(N|Y)) + \text{Var}(E(N|Y))$$
I got stuck on how to calculate $E(N^2)$.
I know that $E(N^2) = E(E(N^2|Y = y))$, but I got stuck onward.
Would greatly appreciate any help!
Ignore the formula $$\operatorname{Var}[N] = \operatorname{E}[N^2] - \operatorname{E}[N]^2.$$ It is the second formula that you want: $$\operatorname{Var}[N] = \operatorname{E}[\operatorname{Var}[N \mid Y]] + \operatorname{Var}[\operatorname{E}[N \mid Y]].$$
How do you use it? What is $\operatorname{E}[N \mid Y]$? It is the expectation of the Poisson distribution with parameter $Y$. What is $\operatorname{Var}[N \mid Y]$? This is the variance of the Poisson distribution with parameter $Y$. So both of these are functions of the random variable $Y$.
Next, compute the expectation and variance of these with respect to the gamma distribution $Y$.
So for example, if $\operatorname{E}[N \mid Y] = Y$, then $$\operatorname{Var}[\operatorname{E}[N \mid Y]] = \operatorname{Var}[Y] = r\lambda^2.$$