What I've tried so far :
$$\int u^m (1-u^2)^n du$$
$$u=\sin x \implies du= \cos x dx$$
$$\int \sin^{m}x \cos^{n+1}x dx$$
I have no clue on how to continue from here.
Also, if the indefinite integral isnt possible to be evaluated. I would like to know how to do a definite integral of this
On
This can be evaluated exactly as a definite integral when the domain is $(0,1)$ via the Beta function. Consider $x=u^2$ (so $dx=2u\,du$). Then, $$ \int_0^1 u^m (1-u^2)^n\,du = \frac{1}{2} \int_0^1 x^{(m-1)/2} (1-x)^n\,dx = \frac{1}{2} B((m+1)/2,n+1). $$
First, use the binomial theorem to expand $(1-u^2)^n$: $$ (1-u^2)^n = \sum_{k = 0}^{n} \binom{n}{k} (-(u^2))^{k} = \sum_{k=0}^{n} \binom{n}{k} (-1)^k u^{2k}. $$ Hence, $\int u^m (1-u^2)^n du = \int \sum_{k = 0}^{n} \binom{n}{k} (-1)^k (u)^{2k+m} du$. Since the summation is finite, we can interchange the integral and the summation to get $$ \sum_{k = 0}^{n} \binom{n}{k} \int (-1)^k (u)^{2k + m} du = \sum_{k=0}^{n} \binom{n}{k} \frac{(-1)^k (u)^{2k+m+1}}{2k+m+1}. $$
I believe that this last series can be expressed more compactly using hypergeometric functions (a la http://www.wolframalpha.com/input/?i=integral+x%5Em+%281-x%5E2%29%5En+dx), but perhaps someone more experience can clarify.