It's not hard to find that $\int\sqrt{\sin^3x - \sin^5x}dx = \frac{2\sin^{5/2}x}{5} + C$,
But the indefinite integral $\int_0^\pi\sqrt{\sin^3x - \sin^5x}dx = \frac{4}{5}$ does not conform to $\cfrac{2\sin^{5/2} \pi}{5} - \cfrac{2\sin^{5/2} 0}{5} = 0 - 0 = 0$
Why is that we can't simply use Newton-Leibniz axiom here and what is the way to find the value $\frac{4}{5}$ here ?
On
Your computation of the indefinite integral is not correct. The square root sign in the integrand is the positve square root. You are assuming that $\cos x$ is positive in your computation. Note that $\cos x$ is positive for $x$ between $0$ and $\frac {\pi } 2$ but negative for $x$ between $\frac {\pi} 2$ and $\pi$.
On
Use $\int_{0}^{2a} f(x) dx=2\int_{0}^{a} f(x) dx....(1)$ if $f(2a-x)=f(x)$, then $$I=\int_{0}^{\pi} \sqrt{\sin^3 x-\sin ^5 x} dx= 2\int_{0}^{\pi/2} \sqrt{\sin^3 x-\sin ^5 x} dx \int_{0}^{\pi/2} (\sin x)^{3/2} \cos x dx$$ $$\implies I =2\int_{0}^{1} t^{3/2} dt=\frac{4}{5}.$$
Note: Always try to reduce the domain of integration if possible by using (1). Other wise $\sin x=t$ is not a good substitution in $(0, \pi)$ because both the limits on $t$ become zero. But it is a good substitution in $(0,\pi/2)$.
I'm going to elaborate on Kavi's answer:
The problem here is that the antiderivative $\frac{2}{5}\sin^{\frac{5}{2}}(x)+C$ is only valid when $x\in\left[0,\frac{\pi}{2}\right]$.
Looking at the Desmos plot I provided, it can be seen that the derivative of $\frac{2}{5}\sin^{\frac{5}{2}}(x)$ does not agree with $\sqrt{\sin^3(x)-\sin^5(x)}$ when $x\in\left(\frac{\pi}{2},\pi\right)$. This is because
\begin{align*} \sqrt{\sin^3(x)-\sin^5(x)} &= \sqrt{\sin^3(x)\left[1-\sin^2(x)\right]}\\ &= \sin^{\frac{3}{2}}(x)\sqrt{\cos^2(x)}\\ &= \sin^{\frac{3}{2}}(x)|\cos(x)| \end{align*}
and $\cos(x)$ is negative when $x\in\left(\frac{\pi}{2},\pi\right)$.
Don't lose hope, though! We can still find the correct antiderivative and apply $\text{FTC}2$. To begin, note that
$$\sqrt{\sin^3(x)-\sin^5(x)}$$
is continuous over $\left[0,\pi\right]$, so $\text{FTC}1$ asserts that the function
$$f(x)=\int_{0}^{x}\sqrt{\sin^3(t)-\sin^5(t)}\text{ }dt$$
is an antiderivative of $\sqrt{\sin^3(x)-\sin^5(x)}$. We can now boil down the problem of finding an antiderivative to finding a nice expression for this integral.
Using the identity $\sqrt{\sin^3(x)-\sin^5(x)}=\sin^{\frac{3}{2}}(x)|\cos(x)|$, we can write
$$f(x)=\int_{0}^{x}\sin^{\frac{3}{2}}(t)|\cos(t)|\text{ }dt$$
so for $x\in\left[0,\frac{\pi}{2}\right]$,
\begin{align*} \int_{0}^{x}\sin^{\frac{3}{2}}(t)|\cos(t)|\text{ }dt &= \int_{0}^{x}\sin^{\frac{3}{2}}(t)\cos(t)\text{ }dt\\ &= \int_{0}^{\sin(x)}u^{\frac{3}{2}}\text{ }du\\ &= \frac{2}{5}\sin^{\frac{5}{2}}(x) \end{align*}
For $x\in\left[\frac{\pi}{2},\pi\right]$,
\begin{align*} \int_{0}^{x}\sin^{\frac{3}{2}}(t)|\cos(t)|\text{ }dt &= \int_{0}^{\frac{\pi}{2}}\sin^{\frac{3}{2}}(t)|\cos(t)|\text{ }dt+\int_{\frac{\pi}{2}}^{x}\sin^{\frac{3}{2}}(t)|\cos(t)|\text{ }dt\\ &= \frac{2}{5}\sin^{\frac{5}{2}}\left(\frac{\pi}{2}\right)-\int_{\frac{\pi}{2}}^{x}\sin^{\frac{3}{2}}(t)\cos(t)\text{ }dt\\ &= \frac{2}{5}-\int_{1}^{\sin(x)}u^{\frac{3}{2}}\text{ }du\\ &= \frac{2}{5}-\left(\frac{2}{5}\sin^{\frac{5}{2}}(x)-\frac{2}{5}\right)\\ &= \frac{4}{5}-\frac{2}{5}\sin^{\frac{5}{2}}(x) \end{align*}
Thus,
$$f(x)=\begin{cases} \frac{2}{5}\sin^{\frac{5}{2}}(x) & \text{if }x\in\left[0,\frac{\pi}{2}\right]\\ \frac{4}{5}-\frac{2}{5}\sin^{\frac{5}{2}}(x) & \text{if }x\in\left[\frac{\pi}{2},\pi\right] \end{cases}$$
We can now (legally) evaluate the original integral with $\text{FTC}2$.
\begin{align*} \int_{0}^{\pi}\sqrt{\sin^3(x)-\sin^5(x)}\text{ }dx &= f(\pi)-f(0)\\ &= \frac{4}{5}-\frac{2}{5}\sin^{\frac{5}{2}}(\pi)-\frac{2}{5}\sin^{\frac{5}{2}}(0)\\ &= \frac{4}{5} \end{align*}