$ I = \int (x^2 + 2x)\cos(x) dx $
Integration by Parts, choose $u$:
$$\begin{align} u &= \cos(x) \\ dv &= (x^2 + 2x)dx \\ du &= -\sin(x) \\ v &= \frac{1}{3}x^3 + x^2 \end{align} $$
Substitute into formula: $$ \begin{align} \int udu &= uv - \int vdu \\ &= \cos(x)\left(\frac{1}{3}x^3 + x^2\right) - \int\left(\frac{1}{3}x^3 + 2x\right)(-\sin(x)) \\ &= \cos(x)\left(\frac{1}{3}x^3 + x^2\right) + \int\left(\frac{1}{3}x^3 + 2x\right)(\sin(x)) \end{align} $$
At this point, it doesn't look like I can use the substitution rule on the the right hand integral, so I decide to use the substitution rule again.
Integration by Parts II, choose $u$:
$$\begin{align} u &= sinx \\ dv &= (\frac{1}{3}x^3 + 2x)dx \\ du &= cosx \\ v &= \frac{1}{12}x^4 + x^2 \end{align} $$
Substitute into formula: $$\begin{align} \int_{}udu &= uv - \int_{}vdu \\ &= (sinx)(\frac{1}{12}x^4 + x^2) - \int_{} (\frac{1}{12}x^4 + x^2)(cosx)dx \end{align} $$
Combining the two integration by parts together and I feel like I am no closer to evaluating the integral than whence I started...The integral is still there and I feel another parts by integration won't work.
$$\int(x^2 + 2x)\cos(x) = (\cos(x))\left(\frac{1}{3}x^3 + x^2\right) + (\sin(x))\left(\frac{1}{12}x^4 + x^2\right) - \int\left(\frac{1}{12}x^4 + x^2\right)(\cos(x))dx$$
Did I do the math wrong and make a mistake somewhere? Or am I supposed to approach this differently?
I would do this way $$\int(x^2+2x)\cos x\ dx$$
By Integration By Parts: $u=(x^2+2x),v^{\prime}=\cos x$ $$=(x^2+2x)\sin x-\int(2x+2)\sin xdx$$
Again apply Integration By Parts for $\int(2x+2)\sin xdx$ $u=(2x+2), v^{\prime}=\sin x$ and we get
$\int(2x+2)\sin xdx=2\sin x-\cos x(2x+2)$
So, we finally get, $$=(x^2+2x)\sin x-[2\sin x-\cos x(2x+2)]+C$$ $$=(x^2+2x-2)\sin x+2(x+1)+C$$
$$\int(x^2+2x)\cos x\ dx=(x^2+2x-2)\sin x+2(x+1)+C$$