Given a real number $\alpha > -1$, a real number $0<\varepsilon<1$, and an integer $n \geq 1$, I can easily compute the following integral by parts:
$$\int_0^{\varepsilon^2} (1-u)^\alpha u^{n-1} \, du.$$
However, I am not able to write the solution in a nice closed form in terms of $n, \alpha, \text{ and } \varepsilon$. Is there a trick to do this?
This is the incomplete beta function, which is defined as
$$\text{B}(\nu,\mu,x)=\int_0^x t^{\nu-1}(1-t)^{\mu-1}~dt,\quad 0\le x\le1,~\nu\gt0,~\mu\gt0$$
Thus, in your case
$$\int_0^{\varepsilon^2} (1-u)^\alpha u^{n-1} \, du=\text{B}(n,\alpha+1,\varepsilon^2)$$
I have verified these results numerically.