How to find coefficient of $x^{12}$ in the expansion of $(1+x+x^2+x^3+...+x^n)^4$
I tried this : Since $(1+x+x^2+x^3+...+x^n)$ is in GP its sum will be $(x^{n+1}+1)(x-1)^{-1}$ now ACQ we have to expand $((x^{n+1}+1)(x-1)^{-1})^4$ next i am not able to do this problem please hep !
Continuing from where you left off but with a corrected formula for the geometric series, use the negative binomial theorem:
$$ (1-x)^{-n}=\sum_{k=0}^\infty \binom{n+k-1}{k} x^k $$
In your case, we have $$ (1-x)^{-4}=\sum_{k=0}^\infty \binom{k+3}{3} x^k = 1+\frac{2\cdot 3 \cdot 4}{1 \cdot 2\cdot 3}x + \frac{3\cdot 4 \cdot 5}{1 \cdot 2\cdot 3}x^2+ \dots $$ and you can then multiply this by $(1-x^{n+1})^4=1-4x^{n+1}+6x^{2n+2}-4x^{3n+3}+x^{4n+4}$ to get your original expression in a more agreeable form. Upon multiplying, we'll get a degree-$12$ contribution for each term this expression, when it's paired off with the corresponding term in the infinite sum such that the total degree of the paired terms is $12$. Thus the coefficient of $x^{12}$ in the product will be
$$ \binom{15}{3}-4 \binom{14 - n}{3}+6 \binom{13 -2n}{3}-4\binom{12-3n}{3}+\binom{11-4n}{3} $$ where we adopt the (somewhat nonstandard) convention that $\binom{n}{k}=0$ for all $n<k$, even $n$ negative.
In particular, whenever $n \geq 12$, the coefficient is always just $\binom{15}{3}=455$.