Find coefficient of $x^ky^m$ in $(x+y-1)^n$

Question

Find coefficient of $x^ky^m$ in $(x+y-1)^n$

I have issue when in polynomial we have $-1$, or another constant.

I will explain, if we have to find coefficient $x^ky^m$ in $(x+y)^n$ then it is easy, according to multinomial theorem. It is $\binom{n}{k,m}$. if we take another example without constant in polynomial, for example to find coefficient of $x^2y^3z^4$ in $(x+y+z)^9$, it is again, according to multinomial theorem $\binom{9}{2,3,4}$.

However, this did not works when we have constant in polynomial, I don't understand how to make this correct here: $\binom{n}{k,m}$ in $(x+y-1)$ and not $(x+y)$.

Answer 1

Now $x^ky^m$ in the expansion of $(x+y-1)^n$ you can see it as $x^ky^m(-1)^{n-(k+m)}$ and therefore its coefficient is given by the multinomial theorem and it's $$ {n}\choose{k,m,n-(m+k)} $$ and whose sign is given by $(-1)^{n-(k+m)}$

Answer 2

Writing $(x+y-1)^n=\sum_{j=0}^n\binom{n}{j}(-1)^{n-j}(x+y)^j$, the $x^ky^m$ contribution is from $j=k+m$ if this is at most $n$, namely$$\binom{n}{k+m}(-1)^{n-k-m}\binom{k+m}{k}=(-1)^{n-k-m}\frac{n!}{k!m!(n-k-m)!}$$in that case, in agreement with @user289143.