How to solve the following indefinite integral $$\int \tan^{3}x \sec^{3/2}x \; dx$$ to get the solution in the form of $$\large\frac{2}{7}\sec^{7/2}x - \frac{2}{3}\sec^{3/2}x +c$$
I tried taking $$u = \sec^{2}x \implies du = \tan x \; dx$$
$$\large \int \tan^{3} x \sec^{\frac{3}{2}}x\;dx = \int (u-1) u ^{\frac{3}{4}}du = \int u^{\frac{7}{4}}- u^{\frac{3}{4}}du = \frac{4}{11} (\sec x)^{\frac{11}{2}} \frac{4}{7} (\sec x)^{\frac{7}{2}} +c$$
Where did I go wrong?
On
While the OP is interested in understanding the source of an error from a specific substitution, I thought that it would be of some interest to show another way to evaluate this indefinite integral.
To that end, we can simply write
$$\tan^3 x\sec^{3/2}x=(1-\cos^2x)\sin x\cos^{-9/2}x$$
so that we have
$$\begin{align} \int\tan^3x\sec^{3/2}x\,dx&=\int\left(\cos^{-9/2}x-\cos^{-5/2}x\right)\sin x\,dx\\\\ &=\frac27 \cos^{-7/2}x-\frac23 \cos^{-3/2}x+C \end{align}$$
Your $du$ is wrong. $(\tan x)'=\sec^2 x$ (not the other way around).
Instead, write the integrand as $\sec x \tan x \tan^2 x \, (\color{maroon}{\sec x})^{1/2}$ and let $u=\color{maroon}{\sec x}$. Write $\tan^2 x$ in terms of $\color{maroon}{\sec x}$ and note $du$ then is $\sec x\tan x\, dx$.