I have to find $a,b \in \mathbb{Z}^*$ such that
$$(1 + x)^5(1+ax)^6 = 1 + bx + 10x^2 + \ldots + a^6x^{11}$$
Now, as far as I can tell for all $a$ there is a solution when $b = 5$, so a solution is $a \in \mathbb{Z}^*$, $b=5$. However, the problem asks for a unique solution. How can there be a unique solution? Or maybe I'm misreading the problem?
We have $$(1+x)^5=1+5x+10x^2+\cdots$$ and $$(1+ax)^6=1+6ax+15a^2x^2+\cdots$$ Multiplying these we get $$(1+x)^5(1+ax)^6=1+(5+6a)x+(10+30a+15a^2)x^2+\cdots$$ Equating coefficients of this polynomial with $1+bx+10x^2+\cdots$ yields the equations $$5+6a=b$$ and $$10+30a+15a^2=10$$ Please let me know if you need assistance in solving these equations for $a,b$.