Given the generating function below, I was struggling trying to come up with a way to get a closed form for the coefficients. $$F(X) = \frac{x^8(x+x^2+x^3)}{(1-x^5)(1-x^2)(1-x)}$$
The question I got this from asked for the closed form of $x_1+5x_2+2x_3+x_4=n$ if $x_4\le3$ and $x_i\ge1$.
I started by trying out partial fraction decomposition, but I'm not sure how to approach it, and there aren't any other avenues I can think of right now. Any advice would be much appreciated!
On
This is more than a hint but less than a solution, though it should be enough to guide you towards one with a bit of work.
Here is a general form of partial fractions that you can use. Suppose that we have a rational function $$R(x)=\frac{f(x)}{g_1(x)g_2(x)\ldots g_n(x)}.$$
Further, suppose that the total degree on top is strictly smaller than the total degree on the bottom, and that all the $g_i(x)$ are relatively prime, that is, they have no factors in common. Then we can find $f_1(x), \ldots, f_n(x)$ with $\deg f_i(x)<\deg g_i(x)$ such that
$$R(x)=\sum_i \frac{f_i(x)}{g_i(x)}.$$
In your particular case, the factors in the bottom all share a factor of $(1-x)$, and so you will have to factor that out and group them all together. Additionally, since the goal is to find a power series, and multiplying a power series by $x^k$ is easy, you might as well factor out $x^9$ from the numerator. This reduces the problem to expressing
$$ \frac{1+x+x^2}{(1-x^5)(1-x^2)(1-x)} = \frac{f_1(x)}{1+x+x^2+x^3+x^4} + \frac{f_2(x)}{1+x}+\frac{f_3(x)}{(1-x)^3}$$ where $\deg f_1(x)\leq 3, \deg f_2(x)=0, \deg f_3(x)\leq 2$.
Multiplying through to clear denominators, this becomes
$$1+x+x^2=f_1(x)(1-x^2)(1-x)^2 + f_2(x)(1-x^5)(1-x)^2 + f_3(x)(1+x+x^2+x^3+x^4)(1+x).$$
Unfortunately, from this point, there is messy algebra to do, plugging in generic polynomials with undetermined coefficients for the $f_i(x)$ and solving. There are a few little tricks you can use to aid your work, but it's going to be rough going in any case, but it is doable. As a hint for this part, I suggest using the fact that if $h(x)=(1-x)^3g(x)$, then $h(1)=h'(1)=h''(1)=0$ in order to find $f_3(x)$ first, and plug in $x=-1$ to find $f_2(x)$. This will leave only $f_1(x)$ unknown, and at this point you can either expand out and compare coefficients or plug in at 4 other random points (such as $0$, $\pm 2$, $3$) to get a system of linear equations to solve for the coefficients. But there are many ways to approach this, so don't feel constrained.
In the final step, when you're looking to get an explicit form of the power series, one nice trick will be to write $$ \frac{f_1(x)}{1+x+x^2+x^3+x^4}=\frac{(1-x)f_1(x)}{1-x^5} $$ and use the fact that $\frac{1}{1-x^5}=\sum_{n=0}^{\infty}x^{5n}.$
Good luck!
Hint
If you start with the long division (forget the $x^8$ for the time being), you will see that the result is in fact $$F(x)=x^9 \sum_{n=0}^\infty a_n\,x^n$$ So, write $$\frac{x^8(x+x^2+x^3)}{(1-x^5)(1-x^2)(1-x)}=x^9 \sum_{n=0}^\infty a_n\,x^n$$ that is to say $$1+x+x^2=(1-x^5)(1-x^2)(1-x)\sum_{n=0}^\infty a_n\,x^n$$ Proceed as usual to find the recurrence relation for the $a_n$'s.