let's say that I got a bin of balls, which its size is distributed negative binomially: NP(2,p). And let's say that each of the balls is numbered from 1 to x-1. What is the generated function of Y: the number of the ball I will have after pulling one random ball from the bin. what's its expected value and varience?
Thanks.
I am assuming $X\sim NB(2,p)$ is the number of balls in the bin and then $Y$ is uniformly picked from the set $\{1,\ldots,X-1\}$ (which are the labels of the balls). Since this selection does not make sense if $X<2$, I am assuming that the convention for the negative binomial distribution $NB(r,p)$ is that its support starts at $r$.
Note from the density of $X$ that $$\mathbb{P}(X-1=k)=\binom{k}{k-1}(1-p)^2p^{k-1}=k(1-p)^2p^{k-1},\quad k\in\{1,2,\ldots\}.$$
Then you may compute, for $t<\log(1/p)$ (so that $pe^t<1$),
$$\begin{split}\mathbb{E}(e^{tY}) &=\sum_{k=1}^\infty \mathbb{E}(e^{tY}|X-1=k)\mathbb{P}(X-1=k)\\ &=\sum_{k=1}^\infty \sum_{n=1}^k e^{tn}\mathbb{P}(Y=n|X-1=k)\mathbb{P}(X-1=k)\\ &=\sum_{k=1}^\infty \sum_{n=1}^k e^{tn}\cdot \frac{1}{k}\cdot k(1-p)^2p^{k-1}\\ &=(1-p)^2\sum_{k=1}^\infty p^{k-1}\sum_{n=1}^k e^{tn} =(1-p)^2\sum_{k=1}^\infty p^{k-1}\frac{e^{t(k+1)}-e^t}{e^t-1}. \end{split}$$
Here we get two geometric series:
$$\begin{split} \sum_{k=1}^\infty p^{k-1}e^{t(k+1)}=e^{2t}\sum_{k=1}^\infty (pe^t)^{k-1}&= \frac{e^{2t}}{1-pe^t},\\ \sum_{k=1}^\infty p^{k-1}&= \frac{1}{1-p}. \end{split}$$
Therefore you finally get, for $t<\log(1/p)$,
$$\mathbb{E}(e^{tY}) =\frac{(1-p)^2}{e^t-1}\sum_{k=1}^\infty p^{k-1}(e^{t(k+1)}-e^t) =\frac{(1-p)^2}{e^t-1}\left(\frac{e^{2t}}{1-pe^t}-\frac{e^t}{1-p}\right). $$