My course suggests to use hyperbolic substitution and the identity $\sinh(2u)=2\sinh(u)\cosh(u)$ for the following integral:
$$ \int x^2 \sqrt{a^2+x^2}dx $$
Let $x=a\sinh(u)$, $dx=a\cosh(u)du$, then:
$$ \begin{align} \int x^2 \sqrt{a^2+x^2}dx = \int a^2 \sinh^2(u) \sqrt{a^2+a^2\sinh^2(u)} \cosh(u) du &= a^4 \int \sinh^2(u) \cosh^2(u) du \end{align} $$
Now I wanted to use the identity $\sinh(2u)=2\sinh(u)\cosh(u)$, so I factored out a $\frac{1}{2}$ to bring the integral into the required form:
$$ \begin{align} a^4 \int \sinh^2(u) \cosh^2(u) du &= a^4 \int \frac{1}{2}(2\sinh(u)\cosh(u)) \frac{1}{2}(2\sinh(u)\cosh(u))du \\ &= \frac{a^4}{4} \int \sinh(2u)^2 du \end{align} $$
I thought this would be the correct replacement, but the solution (problem 5D-6 in the PDF) for this integral shows they arrive at the intermediate step
$$ a^4 \int \sinh^2(u) \cosh^2(u) du = \frac{a^4}{2} \int \sinh(2u)^2 du $$
(Note their $\frac{a^4}{2}$ vs. my $\frac{a^4}{4}$)
Did I make a mistake factoring out the $1/2$ to use the identity? I did the integral twice, but I can't see my mistake.
I don't see anything wrong with what you did. As far as I can tell, the solution set you linked to has a mistake in their answer. Note for their answer to be correct means that
$$\frac{\sinh^2(2y)}{2} = \sinh^2(y)\cosh^2(y) \; \implies \; \sinh(2y) = \pm\sqrt{2}\sinh(y)\cosh(y) \tag{1}\label{eq1}$$
which is not true. Also, it appears it wasn't just a typo in that one place, as this error is carried all the way through to their final answer.