Let $ X,Y $ be 2random variables.
For all $ 1\le m < k $ : $$ P\left(X=k,Y=m\right)=\frac{1}{16}\left(\frac{3}{4}\right)^{k-2}=\frac{1}{9}\left(\frac{3}{4}\right)^{k} $$ Calculate $ Cov(X,Y)$.
Using the formula $$Cov(X,Y)=\mathbb{E}[XY]-\mathbb{E}[X]\mathbb{E}[Y] $$ I tried finding $ \mathbb{E}[XY] $ with $$ \mathbb{E}\left(XY\right)=\sum_{k=1}^{\infty}k\sum_{m=1}^{k}m\cdot\left(\frac{1}{9}\right)\left(\frac{3}{4}\right)^{k}=\sum_{k=1}^{\infty}\frac{1}{2}\left(k^{2}+k^{3}\right)\left(\frac{1}{9}\right)\left(\frac{3}{4}\right)^{k} $$ Looking at the function$$ \frac{1}{1-x}=\sum_{k=0}^{\infty}x^{k}\Rightarrow\left(\frac{1}{1-x}\right)'=\sum_{k=0}^{\infty}kx^{k-1}\Rightarrow\left(\frac{1}{1-x}\right)''=\sum_{k=0}^{\infty}k\left(k-1\right)x^{k-2}\Rightarrow \\ \sum_{k=0}^{\infty}k\left(k-1\right)\left(k-2\right)x^{k-3}=\frac{1}{x^{3}}\sum_{k=0}^{\infty}\left(k^{3}-3k^{2}+2k\right)x^{k}=\left(\frac{1}{1-x}\right)''' \\ \dfrac{x^{3}}{\left(x-1\right)^{4}}=\sum_{k=0}^{\infty}\left(k^{3}-3k^{2}+2k\right)x^{k}\Rightarrow\sum_{k=0}^{\infty}k^{3}x^{k}=\dfrac{x^{3}}{\left(x-1\right)^{4}}-\sum_{k=0}^{\infty}\left(-3k^{2}+2k\right)x^{k} $$ Now I can calculate $ \mathbb{E}[XY] $ ,how do I proceed from here? This seems too complicated for a seemingly elementary covariance calculation, where did I go wrong?
Recognize that $Y|X=k \sim \text{Uniform}\{1,\dots,k-1\}$, therefore $$\mathbb{E}[Y\vert X] = X/2.$$
By the Tower Property, $$\mathbb{E}[XY]=\mathbb{E}[\mathbb{E}[XY\vert X]] = \mathbb{E}[X\mathbb{E}[Y\vert X]]=\frac{1}{2}\mathbb{E}[X^2],$$ and $$\mathbb{E}[Y]=\mathbb{E}[\mathbb{E}[Y\vert X]] = \frac{1}{2}\mathbb{E}[X],$$
so
$$\text{Cov}(X,Y)=\mathbb{E}[YX]-\mathbb{E}[Y]\mathbb{E}[X] = \frac{1}{2}\mathbb{E}[X^2]-\frac{1}{2}\mathbb{E}[X]^2 = \frac{1}{2}\text{Var}(X).$$ Marginally we have $$P(X=k) = \frac{k-1}{16}\left(\frac{3}{4}\right)^{k-2}$$ for $k\in\{2,3,\dots\}.$
You can work out what $\text{Var}(X)$ is like you did above, or recognize $X$ as having the same distribution as a (shifted) negative binomial random variable, i.e. $X=X'+2$, where $X'\sim \text{NB}(r=2, p=\frac{3}{4})$. Therefore $$\text{Cov}(X,Y)=\frac{1}{2}\text{Var}(X) = \frac{1}{2}\text{Var}(X') = \frac{1}{2}\frac{2\cdot\frac{3}{4}}{\left(1-\frac{3}{4}\right)^2}=12.$$