If in the expansion of $(1+x)^m (1-x)^n$, the coefficients of $x$ and $x^2$ are $3$ and $-6$ respectively, then $m$ is
My try: $(1+x)^m (1-x)^n= (m,r)x^r\cdot(n,t)(-x^t)= (m,r)(n,t)(-1)^tx^{r+t}$ Putting $r+t=1$ we get $(m,r)(n,t)(-1)^t=3$. Now $t$ must be equal to $0$,so $r=1$. Therefore, $(m,1)(n,0)=3$ or $m=3$. But the answer given is $12$. Where did I go wrong?
By binomial theorem, $$(1+x)^m=\sum_{k=0}^\infty \binom{m}{k}x^k$$ $$(1-x)^n=\sum_{k=0}^\infty \binom{n}{k}(-x)^k$$ Using convolutions, the coefficient of $x^k$ in $(1+x)^m(1-x)^n$ is $$\sum_{i=0}^k \binom{m}{k-i}\binom{n}{i}(-1)^i$$
So, the coefficient of $x^1$ and $x^2$ are respectively $$\binom{m}{1}-\binom{n}{1}=m-n$$ $$\binom{m}{2}-\binom{m}{1}\binom{n}{1}+\binom{n}{2}=\frac{m^2+n^2-2mn-m-n}{2}$$ Since we are given that the coefficient of $x^1$ is $3$ and the coefficient of $x^2$ is $-6$, we have the following system $$\begin{cases} m-n=3\\m^2+n^2-2mn-m-n=-12\end{cases}$$ $$\begin{cases} m-n=3\\(m-n)^2-m-n=-12\end{cases}$$ $$\begin{cases} m-n=3\\9-m-n=-12\end{cases}$$ $$\begin{cases} m-n=3\\m+n=21\end{cases}$$ $$\begin{cases} \boxed{m=12}\\n=9\end{cases}$$