The question asks me to find the coefficient of the term $x^6y^4$ in the expression $(xy^2+x^2+3y)^7$. This was pretty simple. This is how I did it: $$(xy^2+x^2+3y)^7 = \sum_{a+b+c = n} (xy^2)^a + (x^2)^b + (3y)^c $$ $$ x^{(a+2b)}, y^{(2a+c)}, 3^c = x^6y^4 $$
From this we get, $$ a+2b = 6, 2a+c = 4, a+b+c = 7$$
Solving these will give us $a = 0,b=3, c=4$. After plugging in the values you get 2852 as the final answer.
Now this is where I am stuck, $x^6y^4$ in the expression $(xy^2+x^2+3y+4)^7$. In this question we have an extra term 4 and this is really confusing me. I tried doing it in the same way: $$(xy^2+x^2+3y+4)^7 = \sum_{a+b+c+d = n} (xy^2)^a + (x^2)^b + (3y)^c + 4^d$$ $$ x^{(a+2b)}, y^{(2a+c)}, 3^c, 4^d = x^6y^4 $$ From this we get, $$ a+2b = 6, 2a+c = 4, a+b+c+d = 7$$ I tried solving this but I couldn't get the right answer. Am I doing it correctly. How do I solve it?
You were almost there, mate.
Examine: $a+2b = 6,\, 2a+c = 4,\, a+b+c +d = 7$ for $0\leq a,b,c,d\leq 7$
We have $b=3-a/2$, $c=4-2a, d=7-a-b-c$ so we list the possibilities: $$\begin{cases}a=0, b=3, c=4, d=0 \\[1ex] a=2, b=2, c=0, d=3\end{cases}$$
So the term we require is: $x^{(0+6)}\, y^{(0+4)}\, 3^4\, 4^0 + x^{(2+4)}\, y^{(4+0)}\, 3^0\, 4^3$
Hence the coefficient of $x^6y^4$ is: $3^4+4^3$
That is all.