We can solve the following problem by using substitution and trigonometric identities. Such is the solution: $$ \begin{aligned} & \int \sin ^3 2 \theta \sqrt{\cos 2 \theta} d \theta \\ =& \int \sin ^3 2 \theta(\cos 2 \theta)^{\frac{1}{2}} \\ =& \int \sin ^2 2 \theta \cdot \sin 2 \theta(\cos 2 \theta)^{1 / 2} \\ =& \int\left(1-\cos ^2 2 \theta\right)(\cos 2 \theta)^{1 / 2} \sin 2 \theta d \theta \\ \text { let } & u=\cos 2 \theta \\ & d u=-2 \sin 2 \theta d \theta \\ & d \theta=-\frac{d u}{2 \sin 2 \theta} \\ =& \int\left(1-u^2\right)(u)^{1 / 2} \cdot \sin 2 \theta \cdot \frac{d u}{2 \sin 2 \theta} \\ =&-\frac{1}{2} \int\left(1-u^2\right)(u)^{1 / 2} d u \\ =&-\frac{1}{2} \int\left(u^{1 / 2}-u^{3 / 2}\right) d u \\ &=-\frac{1}{2}\left[\frac{2}{3} u^{3 / 2}-\frac{2}{5} u^{5 / 2}\right] \\ &=-\frac{1}{3} u^{3 / 2}+\frac{1}{5} u^{5 / 2}+C \\ &=-\frac{1}{3} \cos ^{3 / 2} 2 \theta+\frac{1}{5} \cos ^{5 / 2} 2 \theta+C \end{aligned}$$
We are trying to find out how to solve the problem using Integration by Parts (The one where it says $\int udv = uv - \int vdu$). We have arrived at the solution below but we do not know how to continue from here until we get the desired result which is $-\frac{1}{3} \cos ^{3 / 2} 2 \theta+\frac{1}{5} \cos ^{5 / 2} 2 \theta+C$. Any help would be appreciated.
$$ \begin{aligned} &\int \sin ^3 2 \theta \sqrt{\cos 2 \theta} d \theta \\ &\int \sin ^2 2 \theta \sin 2 \theta \sqrt{\cos 2 \theta} d \theta \\ &\text { Let } u=\sin ^2 2 \theta, d v=\sin 2 \theta \sqrt{\cos 2 \theta} d \theta \\ &d u=\sin 4 \theta d \theta, \quad v=\int \sin 2 \theta \sqrt{\cos 2 \theta} d \theta \\ &\text { Let } x=\cos 2 \theta \\ &d x=-2 \sin 2 \theta d \theta \\ &\therefore d \theta=\frac{d x}{-2 \sin 2 \theta} \\ &\therefore v=-\frac{1}{2} \int \sqrt{x} d x \\ &=-\frac{1}{2}\left[\frac{2}{3} u^{3 / 2}\right]=-\frac{1}{3} \cos ^{3 / 2} 2 \theta \\ &\therefore \sin ^2 2 \theta\left(-\frac{1}{3} \cos ^{3 / 2} 2 \theta\right)-\int\left(-\frac{1}{3} \cos ^{3 / 2} 2 \theta\right) \cdot \sin 4 \theta d \theta \\ & \end{aligned} $$
There are a couple of mistakes in your solutions:
Since $\sin(2x) = 2\sin(x) \cos(x)$ we get $$ \int \cos^{\frac32}(2\theta) \sin(4\theta) \, \mathrm{d}\theta = 2\int \cos^{\frac52}(2\theta) \sin(2\theta)\, \mathrm{d} \theta \overset{\color{blue}{u = \cos(2\theta)}}{=} - \int u^{\frac52}\, \mathrm{d}u = - \frac27 \cos^{\frac72}(2\theta) \tag{1} $$ Then, from your last (corrected) equation, we get \begin{align*} \int\sin^3(2\theta)\sqrt{\cos(2\theta)}\, \mathrm{d} \theta & \overset{\text{IBP}}{=} -\frac{\color{purple}{\sin^2(2\theta)}\cos^{\frac32}(2\theta)}{3} + \frac{\color{green}{2}}{3} \int \cos^{\frac32}(2\theta) \sin(4\theta) \, \mathrm{d}\theta \\ & \overset{(1)}{=} -\frac{\color{purple}{\left(1-\cos^2(2\theta)\right)}\cos^{\frac32}(2\theta)}{3} - \frac{4}{21} \cos^{\frac72}(2\theta) +C\\ & = -\frac{1}{3}\cos^{\frac32}(2\theta) + \frac{1}{7}\cos^{\frac72}(2\theta)+C \end{align*} as desired.