I need to find $[x^3](1-x)^{-1}(1-2x)^6$, where $[x^3]$ means the coefficent of the $[x^3]$ term. here's what I've done:
$[x^3](1-x)^{-1}(1-2x)^6=[x^3](\sum_{k=0}^6 {6\choose k}(-2x)^k)(\sum_{m=0}^\infty {m\choose 0}x^m)$
$= \sum_{k=0}^6 {6\choose k}(-2)^k[x^{3-k} ](\sum_{m=0}^\infty {m\choose 0}x^m)$
$= \sum_{k=0}^3 {6\choose k}(-2)^k[x^{3-k} ](\sum_{m=0}^\infty {m\choose 0}x^m)$ since we need $3-k \geq 0$
$= \sum_{k=0}^3 ({6\choose k}(-2)^k {3-k\choose 0})$
$= \sum_{k=0}^3 ({6\choose k}(-2)^k)$
$= {6\choose0} + (-2){6\choose1} + (4){6\choose2} + (-8){6\choose3}$
$=1-12+60-160$
$= -111$
But when I do the expansion on WolframAlpha, I see that $[x^0]=1$, $[x^1]=-12$, $[x^3]=-160$, so what am I doing wrong?
(I am following a similar idea to Trevor Gunn's answer in this question In how many ways the sum of 5 thrown dice is 25?)
As Mike Earnest confirmed in the comments, your work is correct. As I commented, and you've stated it's likely the case, the WolframAlpha results of $[x^0]=1$, $[x^1]=-12$, $[x^3]=-160$ probably come from the coefficients in the power expansion of $(1-2x)^6$ instead. You can see this directly from the first, second & fourth terms in your second last highlighted line, i.e.,