One of the problems in my Discrete Math course states that we need to find the coefficient of $x^k$ in generating function $f(x)=\frac{1}{(1-x)^3(1+x)}$. The answer is $$\frac{1}{8}+\frac{1}{4}(k+1)+\frac{1}{2}\binom{k+2}{2}+\frac{(-1)^k}{8}$$ The solution was using partial fraction decomposition. I wanted to use the identities for the formal power series. For example: $$\frac{1}{1+x}=\frac{1}{1-(-x)}=\sum_{k=0}^\infty \binom{n-1+k}{k}(-1)^kx^k$$
If $x^k=x^0$ then: $$\binom{1-1+0}{0}x^0=\binom{0}{0}x^0=1$$ If $x^k=x^1$ then: $$-\binom{1-1+1}{1}x^1=-\binom{1}{1}x^1=-1$$ Therefore since: $$\frac{1}{(1-x)^3}=\sum_{k=0}^\infty \binom{n-1+k}{k}x^k$$ Hence the coefficient of $x^k$ would be: $$\binom{0}{0}x^0\binom{3-1+k}{k}x^k-\binom{1}{1}x^1\binom{3-1+k-1}{k-1}x^{k-1}=$$ $$=\binom{k+2}{k}-\binom{k+1}{k-1}$$ When check my solution it doesn't equal to the textbook's solution. I really want to use identities for such problems, can I still use them and if yes what is the correct identity or what am I doing wrong?
We have
$$\frac1{1+x}=\sum_{n\ge 0}(-1)^nx^n$$
and
$$\frac1{(1-x)^3}=\sum_{n\ge 0}\binom{n+2}2x^n\;,$$
so
$$\frac1{(1+x)(1-x)^3}=\left(\sum_{n\ge 0}(-1)^nx^n\right)\left(\sum_{n\ge 0}\binom{n+2}2x^n\right)\;.\tag{1}$$
It’s at this point that you went astray. The coefficent of $x^n$ in $(1)$ is
$$\begin{align*} \sum_{k=0}^n(-1)^k\binom{(n-k)+2}2&=\sum_{k=0}^n(-1)^k\binom{n+2-k}2\\ &=\sum_{\ell=2}^{n+2}(-1)^{n+2-\ell}\binom{\ell}2&&\text{setting }\ell=n+2-k\\ &=(-1)^n\sum_{\ell=2}^{n+2}(-1)^\ell\binom{\ell}2\;, \end{align*}$$
which is probably not something for which you know a closed form. It turns out that
$$\begin{align*} (-1)^n\sum_{\ell=2}^{n+2}(-1)^\ell\binom{\ell}2&=(-1)^n(-1)^n\left\lfloor\frac{(n+2)^2}4\right\rfloor\\ &=\left\lfloor\frac{(n+2)^2}4\right\rfloor\;. \end{align*}$$
It is not obvious that this is equal to
$$\frac18+\frac14(n+1)+\frac12\binom{n+2}2+\frac{(-1)^n}8\;,$$
but in fact
$$\begin{align*} \frac{1+(-1)^n}8+\frac{n+1}4+\frac12\binom{n+2}2&=\frac{1+(-1)^n}8+\frac{(n+1)(n+3)}4\\ &=\frac{1+(-1)^n}8+\frac{(n+2)^2-1}4\\ &=\frac{(n+2)^2}4+\frac{(-1)^n-1}8\\ &=\begin{cases} \frac{(n+2)^2}4,&\text{if }n\text{ is even}\\ \frac{(n+2)^2-1}4,&\text{if }n\text{ is odd} \end{cases}\\ &=\left\lfloor\frac{(n+2)^2}4\right\rfloor\;. \end{align*}$$