Let's consider a Bernoulli trial where $p$ denotes the probability of success. A random variable $X$ that counts the frequency of failures until the $r$-th success has a negative binomial distribution. We can assume that $$P(\{X=k\})=p^r(1-p)^{k}{r+k-1\choose k}$$ and $$\mathbb{E}(X)=\frac{r(1-p)}{p}.$$ Compute $\mathbb{E}(X(X-1))$. (We can assume that this expression is well defined)
My approach:
\begin{align*} &\mathbb{E}(X(X-1))=\sum\limits_{k=1}^{\infty}k(k-1)p^r(1-p)^{k}{r+k-1\choose k}=\frac{r(1-p)}{p}\sum\limits_{k=1}^{\infty}(k-1)p^{r+1}(1-p)^{k-1}\frac{(k+r-1)!}{r!(k-1)!}\\ &=\frac{r(1-p)}{p}\sum\limits_{k=1}^{\infty}(k-1)p^{r+1}(1-p)^{k-1}{(r+k-1)\choose k-1}=\frac{r(1-p)}{p}\sum\limits_{k=0}^{\infty}kp^{r+1}(1-p)^{k}{(r+k)\choose k}=\dots. \end{align*}
Now we see that $\sum\limits_{k=0}^{\infty}kp^{r+1}(1-p)^{k}{(r+k)\choose k}$ is the expected value of another random variable $Y$ that has a negative binomial distribution. But $Y$ counts the failures until the $r+1$-th success and so $\mathbb{E}(Y)=\frac{(r+1)(1-p)}{p}$. Hence,
$ \begin{align*} \dots=\frac{r(1-p)}{p}\mathbb{E}(Y)=\frac{r(1-p)}{p}\frac{(r+1)(1-p)}{p}=\frac{r(r+1)(1-p)^2}{p^2}. \end{align*}$
The sample solution is $\frac{r}{p}\left(\frac{r+1}{p}-2\right)$. So I assume that my approach must be flawed somewhere. However, I dont see where I messed it up!?
Can someone help me?
I can confirm your answer is correct. $$ \begin{align} E[X(X-1)] &=\sum_{k=2}^\infty p^r(1-p)^k\cdot \color{blue}{k(k-1)\binom{r-1+k}{k}} \\&=\sum_{k=2}^\infty p^r(1-p)^k\cdot \color{blue}{(r+1)r\binom{r-1+k}{k-2}} \\&=(r+1)r\cdot \frac{(1-p)^2}{p^2}\sum_{k=2}^\infty p^{r+2}(1-p)^{k-2}\binom{r-1+k}{k-2} \\&=(r+1)r\cdot \frac{(1-p)^2}{p^2}\sum_{k=0}^\infty p^{r+2}(1-p)^{k}\binom{(r+2)-1+k}{k} \\&=(r+1)r\cdot \frac{(1-p)^2}{p^2}\cdot 1 \end{align} $$ The final summation is $1$, because it is the total probability for a negative binomal distribution with parameters $r+2$ and $p$.
In general, $$ E[X(X-1)\cdots (X-n+1)]=r(r+1)\cdots(r+n-1)\left(\frac{1-p}p\right)^n $$