Which sequence is generated by $\frac{{5x - 3{x^2}}}{{{{(1 - x)}^3}}}$?
We know that:
$$\frac{1}{{{{(1 - x)}^3}}} = \sum\limits_{j = 0}^\infty {\left( {\begin{array}{*{20}{c}}
{j + 2} \\
2 \\
\end{array}} \right){x^j}} $$
So we have:
$$(5x - 3{x^2}) \cdot \sum\limits_{j = 0}^\infty {\left( {\begin{array}{*{20}{c}} {j + 2} \\ 2 \\ \end{array}} \right){x^j}} $$
Now it's easy to see that for $x^k$, $a_k$ can be defined by:
$$5\left( {\begin{array}{*{20}{c}} {k + 1} \\ 2 \\ \end{array}} \right) - 3\left( {\begin{array}{*{20}{c}} k \\ 2 \\ \end{array}} \right)$$
Because we "used" $x^{k-1}$ and $x^{k-2}$ there may be a problem for $k=0,1$ and I've been told I need to check it directly. Can you help me with that?
I think you're overthinking things. First of all, \begin{align*} 5 {k + 1 \choose 2} - 3{k \choose 2} &= 5\frac{(k+1)k}{2} - 3\frac{k(k-1)}{2} \\ &= \frac{5}{2} k^2 + \frac52 k - \frac32k^2 + \frac32 k \\ &= 4k + k^2 \\ &= k(k + 4) \\ \end{align*}
So your sequence is $a_k = k(k+4)$. Second of all, they probably want you to plug in $k = 0$ and $k = 1$, giving $$ a_0 = 0, a_1 = 5 $$ Now check that these particular cases line up with the series expression you had before $$(5x - 3{x^2}) \cdot \sum\limits_{j = 0}^\infty {\left( {\begin{array}{*{20}{c}} {j + 2} \\ 2 \\ \end{array}} \right){x^j}}. $$
That is, check that the series above starts out with $0 + 5x^1 + \cdots$.