I need to integrate the following using substitution:
$$ \int x^2\sqrt{x^2+1}\;dx $$
My textbook has a similar example:
$$ \int \sqrt{x^2+1}\;x^5\;dx $$
They integrate by splitting $x^5$ into $x^4\cdot x$ and then substituting with $u=x^2+1$:
$$ \int \sqrt{x^2+1}\;x^4\cdot x\;dx\\ =\frac{1}{2}\int \sqrt{u}\;(u-1)^2\;du\\ =\frac{1}{2}\int u^{\frac{1}{2}}(u^2-2u+1)\;du\\ =\frac{1}{2}\int u^{\frac{5}{2}}-2u^{\frac{3}{2}}+u^{\frac{1}{2}}\;du\\ =\frac{1}{7}u^{\frac{7}{2}}-\frac{2}{5}u^{\frac{5}{2}}+\frac{1}{3}u^{\frac{3}{2}}+C $$
So far so good. But when I try this method on the given integral, I get the following:
$$ \int x^2\sqrt{x^2+1}\;dx\\ =\frac{1}{2}\int \sqrt{x^2+1}\;x\cdot x\;dx\\ =\frac{1}{2}\int \sqrt{u}\;\sqrt{u-1}\;du\;(u=x^2+1)\\ =\frac{1}{2}\int u^{\frac{1}{2}}(u-1)^\frac{1}{2}\;du $$
Here is where it falls down. I can't expand the $(u-1)^\frac{1}{2}$ factor like the $(u-1)^2$ factor above was, because it results in an infinite series. I couldn't prove, but I think any even exponent for the $x$ factor outside the square root will cause an infinite series to result. Odd exponents for $x$ will work, since it will cause the $(u-1)$ term to have a positive integer exponent.
How should I proceed? I don't necessarily want an answer. I just want to know if I'm missing something obvious or if it is indeed above first year calculus level and probably a typo on the question.
Proceeding with your method
$\begin{align} \int x^2\sqrt{x^2+1}\;dx &=\int \sqrt{x^2+1}\;x\cdot x\;dx\\ &=\frac1{2}\int \sqrt{u}\;\sqrt{u-1}\;du\;\left( \text{ let $\begin{align} u&=x^2+1 \\ du&=2xdx\end{align}$}\right)\\ &=\frac1{2}\int \sqrt{u^2-u} \;du\\ &=\frac1{2} \int \sqrt{\left(u-\frac1{2}\right)^2-\left(\frac1{2}\right)^2} \;du \;\text{ (completing the square)} \\ &=\frac1{2}\left({\left(u-\frac1{2}\right)\over 2}{\sqrt{u^2-u}}-{\left(\frac1{2}\right)^2 \over 2}{ \ln \left|\left(u-\frac1{2}\right) +\sqrt{u^2-u}\right|} +C\right)\\ &\text{ (using $\int \sqrt{x^2-a^2} dx={x\over 2}{\sqrt{x^2-a^2}}-{a^2 \over 2}{ \ln |x+\sqrt{x^2-a^2}|} +C)$}\\ &={(2u-1)\over 8}{\sqrt{u^2-u}}-{\frac{ \ln |(2u-1) +2 \sqrt{u^2-u}|}{16}} +C'\\ &\text{ substituting $\left(\begin{align} u&=x^2+1 \\ 2u-1 &=2x^2 + 1 \\ \sqrt{u^2-u}&=\sqrt{(x^2 +1)^2-(x^2+1)}=\sqrt{x^4+x^2}\end{align}\right)$} \\ &=\frac{{(2x^2 + 1)}{\sqrt{x^4+x^2}}}{8}-{\frac{ \ln |(2x^2+1) +2 \sqrt{x^4+x^2}|}{16}} +C'\\ &=\frac{{(2x^3 + x)}{\sqrt{x^2+1}}}{8}-{\frac{ \ln |(x^2+(x^2+1) +2x \sqrt{x^2+1}|}{16}} +C'\\ &=\frac{{(2x^3 + x)}{\sqrt{x^2+1}}}{8}-{\frac{ \ln |(x+\sqrt{x^2+1})^2|}{16}} +C'\\ &=\frac{{(2x^3 + x)}{\sqrt{x^2+1}}-\ln |x+\sqrt{x^2+1}|}{8} +C' \end{align}$