I would like to find a closed form for the following integral:
$$I=\int_0^1 d\alpha\ \alpha^{\omega-5/2} (1-\alpha)^{-1/2} \int_0^\alpha d\beta\ \beta^{2\omega-3} (1-\alpha-\beta)^{5/2-2\omega} (\alpha + \beta)^{9/2-3\omega} \tag{1}$$
I am interested in the case $\omega := \epsilon + 2 \to 2$, where I expect the integral to be divergent. Hence I expect an expression of $\Gamma$ functions, with at least one of them being of the form $\Gamma(\epsilon)$ after expansion around $\epsilon \sim 0$.
Here is what I tried. First I rewrote the $\beta$-integral with the following substitution $\beta \to \beta/\alpha$ such that the limits of integration are $0$ and $1$. This step might be critical (is it?), since $\alpha$ can take the value $0$. Rearranging the terms in a convenient way, the integral can be rewritten as an Appell function:
$$\begin{align} I &= \int_0^1 d\alpha\ (1-\alpha)^{2-2\omega} \int_0^1 d\beta\ \beta^{2\omega-3} (1-(-1)\beta)^{9/2-3\omega} \left(1- \frac{\alpha}{1-\alpha}\beta\right)^{5/2-3\omega} \\ &= \frac{\Gamma(2\omega-2)}{\Gamma(2\omega-1)} \int_0^1 d\alpha\ (1-\alpha)^{2-2\omega} F_1 \left(2\omega-2,2\omega-5/2,3\omega-9/2,2\omega-1 \left| \frac{\alpha}{1-\alpha},-1 \right.\right) \tag{2} \end{align}$$
Using the series representation of the Appell function, I can do the last integral:
$$\begin{align} I &= \frac{\Gamma(2\omega-2)}{\Gamma(2\omega-1)} \sum_{m,n=0}^\infty (-1)^n \frac{\Gamma(2\omega-2+m+n) \Gamma(2\omega-5/2+m) \Gamma(3\omega-9/2+n) \Gamma(2\omega-1)}{\Gamma(2\omega-2)\Gamma(2\omega-5/2) \Gamma(3\omega-9/2) \Gamma(2\omega-1+m+n) \Gamma(m+1) \Gamma(n+1)} \\ &\qquad \qquad \times \int_0^1 d\alpha\ \alpha^m (1-\alpha)^{2-2\omega-m} \\ & = \frac{\Gamma(2\omega-2)}{\Gamma(2\omega-1)} \sum_{m,n=0}^\infty (-1)^n \frac{\Gamma(2\omega-2+m+n) \Gamma(2\omega-5/2+m) \Gamma(3\omega-9/2+n) \Gamma(2\omega-1)}{\Gamma(2\omega-2)\Gamma(2\omega-5/2) \Gamma(3\omega-9/2) \Gamma(2\omega-1+m+n) \Gamma(m+1) \Gamma(n+1)} \\ &\qquad \qquad \times \frac{\Gamma(m+1) \Gamma(3-2\omega-m)}{\Gamma(4-2\omega)} \tag{3}\end{align}$$
where in the second equality I have used the relationship between beta function and gamma function. This step may also be critical, since for $m=0$ the integral diverges, while $\Gamma(3-2\omega)/\Gamma(4-2\omega)$ does not. In general I am a little confused, since it seems that the divergence fully disappeared in my final result (all the $\Gamma$-functions are finite for all $m$!)
So the question is clear: where is(are) my mistake(s), and what is the right way to perform this integral?
Many thanks in advance!
Can you double-check whether the equation (1) is a correct starting point? The factor $(1-\alpha-\beta)^{5/2-2\omega}$ has a branch point at $\beta=1-\alpha$ which is in the integration range if $\alpha>1/2$. Is it really what you had in mind? I think no since the internal $\beta$ integral diverges near the branch point even before $\epsilon=0$ is approached.