Find coefficient of $x^{10}$ in $x^2(x^2-3x^3-1)^6$

Question

I'm trying to solve the following problem (final exams preparation):

Find coefficient of $x^{10}$ in $x^2(x^2-3x^3-1)^6$.

The steps I have taken so far:

$x^2(x^2-3x^3-1)^6 = $

$=x^2\sum \binom{6}{n1,n2,n3}(x)^{2n_1}(-3)^{n_2}(x)^{3n_2}(-1)^{n_3}$

$n_1+n_2+n_3=6$

$2n_1+3n_2=8$

$\Rightarrow$ $n_1=1 \wedge n_2=2$

$\Rightarrow$ $n_1=4 \wedge n_2=0$

Are my steps correct? If so, how do I continue?

Thanks

Answer 1

I'll translate the work you did into English.

So the coefficient of the $x^{10}$ term in $x^2(x^2-3x^3-1)^6$ is the same as the coefficient of the $x^8$ term in $(x^2-3x^3-1)^6$. To figure out which terms in that trinomial power contribute that that term, we effectively want to find the non-negative solutions to $2a+3b+c=8$ and $a+b+c=6$. As you note, the solutions to that are $(a,b,c)\in\{(4,0,2),(1,2,3)\}$.

Our last step is appealing to the multinomial formula to find those two terms. They are $$\frac{6!}{4!2!}(x^2)^4(-3x^3)^0(-1)^2=15x^8\\\frac{6!}{3!2!}(x^2)^1(-3x^3)^2(-1)^3=90\cdot x^2\cdot9x^6\cdot(-1)=-540x^8$$ which lead to a total coefficient of $15-540=-525$.