Well my actual question was:
Q1. How do I find coefficient of $x^4$ in $(1+x+x^2)^3$
Of course I can enumerate the combinations and quickly realized that its 6: $$\{x.x.x^2,x^2.x.x,x.x^2.x,x^2.x^2.1,1.x^2.x^2,x^2.1.x^2\}$$
But I want to know if there are any neat set of steps with which I can come up with straight combinatorial formula (as I do in below problem), so that I can prepare such formula for higher power problems as stated in the title of the post.
If I take the problem:
Find coefficient of $x^2$ in $(1+x+x^2)^4$
Its easy. I will find that, we have to find coefficient of $x^2$. Here the power is 2 which same as biggest power in series $1+x+x^2$. So instead of $(1+x+x^2)^4$, I will consider $(1+x+x^2+...)^4$
$$(1+x+x^2+...)^4=\frac{1}{(1-x)^4}=\sum_{k=0}^\infty \binom{3+k}{3}x^k$$
So I have to find coefficient of $x^k$ for $k=2$ which is $\binom{3+2}{3}=10$.
However in Q1., power 4 of $x^4$ is bigger than the max power 2 in series. So I cannot consider infinite series (instead of finite) as it will add additional combinations such as $x.1.x^3$
My textbook gives tricky solution for this, such as dividing and multiplying the series with $(1-x)^3$ (which I dont find intuitive) and doing further unintuitive adjustments.
Is there any intuitive or more importantly, works-for-all-problems approach?
$(1+x+x^2)^3 = \left(\frac{1-x^3}{1-x}\right)^3 = (1-x^3)^3 \frac{1}{(1-x)^3} = (1-3x^3 + 3x^6 - x^9)(\sum_{n=}^\infty \binom{n+2}{2}x^n)$ and so we can make the coefficient of $x^4$ from $1 \cdot \binom{4+2}{2}$ and $-3\binom{2+1}{2}$ and this equals $15 - 3\cdot 3 = 6$