How many ways are there to put 8 identical balls into 3 boxes so that no box has more than 4 balls in it? But Box 3 can only have up to 2 balls inside it. I started the problem by trying to write out the polynomials. For the first two boxes I thought it would be (1 + x + x ² + $x^3$ + $x^4$ )² and then for Box 3 it would be (1 + x + x ²) So altogether, (1 + x + x ²)(1 + x + x ² + $x^3$ + $x^4$ )² But from here, I am unsure how to write it as a series to find the number of ways for 8 balls.
A computer multiplied it and got $1 + 3 x + 6 x^2 + 9 x^3 + 12 x^4 + 13 x^5 + 12 x^6 + 9 x^7 + 6 x^8 + 3 x^9 + x^{10} + 0+0+ \ldots$
Your generating function approach to giving the answer $6$ as the coefficient of $x^8$ in the expansion of $(1+x+x^2)(1+x+x^2+x^3+x^4)^2$ is correct and the possibilities are:
Your method assumes the balls are indistinguishable but the boxes are distinguishable. Other assumptions would produce different results.