Use a generating function to determine the number of ways to select a hand of m cards from a triple deck, if there are n distinct cards in a single deck. Verify that your expression produces the correct answers when n = 52 and m = 5 or m = 6.
I know that if m = 5 there are 3,817,112 possibilities and if m = 6 there are 36,216,596 possibilities. I also know $G_n(x) = (1+x+x^2+x^3)^{52}$ is the generating function. The only possible way I have seen to solve this problem is by applying the binomial theorem thrice. However, I am having trouble simplifying it all the way. Here is what I have so far: $$G_n(x) = (1+(x+x^2+x^3))^{52} = \sum_{k=0}^{52}\binom{52}{k}(x+x^2+x^3)^k=\sum_{k=0}^{52}\binom{52}{k}x^k(1+(x+x^2))^k=\sum_{k=0}^{52}\binom{52}{k}x^k\sum_{j=0}^{k}\binom{k}{j}(x+x^2)^j=\sum_{k=0}^{52}\binom{52}{k}x^k\sum_{j=0}^{k}\binom{k}{j}x^j(1+x)^j=\sum_{k=0}^{52}\binom{52}{k}x^k\sum_{j=0}^{k}\binom{k}{j}x^j\sum_{l=0}^j\binom{j}{l}x^l=\sum_{k=0}^{52}\sum_{j=0}^k\sum_{l=0}^j\binom{52}{k}\binom{k}{j}\binom{j}{l}x^{l+j+k}$$
I am very confused from this point on. If anyone could shed some light on this it would be greatly appreciated.
Thank you in advance and let me know if there is anything unclear.
There is a simpler approach. Rewrite the generating function $G_n(x)$ as $$ G_n(x)=(1+x+x^2+x^3)^n = \left(\frac{1-x^4}{1-x}\right)^n=(1-x^4)^n(1-x)^{-n} $$ We want the $x^m$ coefficient of $G_n(x)$. Since $G_n(x)$ is the product of $(1-x^4)^n$ and $(1-x)^{-n}$, we can find this coefficient by finding the series these two are generating functions of, $$ (1-x^4)^n=\sum_{i\ge0}\binom{n}{i}(-1)^i x^{4i} $$ $$ (1-x)^{-n}=\sum_{j\ge0}\binom{-n}{j}(-1)^jx^j=\sum_{j\ge0}\binom{n+j-1}{j}x^j $$ then taking the convolution of these two series: $$ [x^m]G_n(x) = \sum_{k\ge0} (-1)^k\binom{n}{k}\binom{n+(m-4k)-1}{m-4k} $$