How to find the coefficient of $x^3y^5$ in the expression $(1+xy+y^2)^n$

Question

We are given the expression $(1+xy+y^2)^n$ where $n$ is positive integer. We are required to find the coefficient of $x^3y^5$ in the expansion of the given expression.

I know how to expand a binom to certain power, if it was $(xy+y^2)^n$ then $n = 4$ and the coefficient would be $\binom{4}{1}(xy)^3(y^2)^1$, but I'm not sure how to solve this when we have expression with three terms.

Answer 1

We may have

• $x^3y^5=(xy)^3y^2$

then by trinomial expansion

$$(1+xy+y^2)^n =\ldots+ \frac{5!}{1!2!3!}(xy)^3y^2+\ldots$$

Answer 2

The coefficient of $x^3$ in $$(xy+(1+y^2))^n$$ will be $$\binom n3y^3(1+y^2)^{n-3}$$

So, we need the coefficient of $y^{5-3}$ in $$(1+y^2)^{n-3}$$ which is $\binom{n-3}1$