I am trying to solve the following:
Let $X_{i}$~$Geometric(q) i=1,2,...,N$ with $q=1-p, 0<p<1$. $N$~$Geometric(p)$. Define $Y=\sum_{i=1}^{N}X_i$ and assume each $X_i$ is i.i.d. and independent from $N$. Using the probability generating function derive $P(Y=k), k=0,1,2,...$.
My attempt so far: Using the properties of random sums of random variable, we know the probability generating function for $Y, g_Y(t)=g_N(g_X(t)).$ Since $N$ and $X$ are both distributed geometrically we have, $g_N(t)=\frac{q}{1-tp}$ since $N$ is $Geometric(q)$ and $g_X(t)=\frac{p}{1-tq}$ since $X$ is $Geometric(p)$. Then $g_Y(t)=g_N(g_X(t))=\frac{q}{1-(\frac{p}{1-tq})p}=\frac{q(1-qt)}{p^2+qt-1}$. From here I don't know how to reduce from the probability generating function of $Y$ to the distribution of $Y$.
Using the fact that $P(X=k)=\frac{g^{(k)}(0)}{k!}$, where $g^{(k)}$ is the kth derivative of the generating function. Taking successive derviatives I find that $g_{Y}'(t)=\frac{pq^3}{(pq+qt-1)^2}\\g_{Y}''(t)=\frac{-2pq^4}{(pq+qt-1)^3}\\g_{Y}'''(t)=\frac{6pq^5}{(pq+qt-1)^4}\\g_{Y}''''(t)=\frac{-24pq^6}{(pq+qt-1)^5}$.
This leads me to believe that the kth derivative of the generating function is of the form $g_{Y}^{(k)}(0)=\frac{(-1)^{k+1}k!pq^{k+2}}{(pq-1)^{k+1}}$ for $k=1,2,...$, thus $P(X=k)=\frac{(-1)^{k+1}pq^{k+2}}{(pq-1)^{k+1}}$ But what do I do when $k=0$? When $k=0$ the numerator looks totally different, there is no factor of p...
You have a minor issue with the expression for $g_Y$:
$$g_Y(t)=\frac{q(qt-1)}{p^2+qt-1}$$
Then
$$P\{Y=k\}=\frac{g_Y^{(k)}(0)}{k!}=(-1)^{k+1}\frac{(q-1)^2}{(q-2)^{k+1}} \text{, }k=0,1,\dots$$