I want to express \begin{align} \int_0^1 dx \phantom{1} (1-x)^{\alpha} x^{\beta} \left( (1-x)^2 a + x^2 b \right)^{-\frac{\alpha+\beta}{2}} \end{align} In terms of products of Gamma function.
Recall the Beta function \begin{align} B(x,y) = \int_0^1 t^{x-1} (1-t)^{y-1} dt = \frac{\Gamma(x)\Gamma(y)}{\Gamma(x+y)} \end{align} The shape is similar except the additional factor $((1-x)^2 a + x^2 b)$....
I can’t answer your question in general, and doubt a general answer exists: but, I can answer a special case.
$\newcommand{\d}{\,\mathrm{d}}\newcommand{\res}{\operatorname{Res}}$Suppose $\alpha,\beta>-1$ and $\alpha+\beta=2m$ where $m\in\Bbb N$ ($m\ge0$ is guaranteed by the conditions on $\alpha,\beta$ and $m=0$ is just handled by the Beta function ordinarily). Suppose also that $a,b>0$. Suppose for now that $\beta$ itself (and thus also $\alpha$) is not an integer - the case where they are (nonnegative) integers follows either from more elementary techniques or from taking limits.
Let $\log_1:\Bbb C^\star\to\Bbb C$ be the logarithm induced by $0\le\arg<2\pi$ and $\log_2:\Bbb C^\star\to\Bbb C$ be the logarithm induced by $-\pi\le\arg<\pi$.
Define: $$f:\Bbb C\setminus[0,1]\to\Bbb C$$Through: $$z\mapsto\exp(\alpha\log_2(1-z))\exp(\beta\log_1(z))\cdot((1-z)^2a+z^2b)^{-m}$$
This is meromorphic (! this crucially relies on $\alpha+\beta\in\Bbb Z$) with poles at the roots of $(1-z)^2a+z^2b=z^2(a+b)-2za+a$, which occur at:
$$\frac{2a\pm\sqrt{4a^2-4a(a+b)}}{2(a+b)}=\frac{a\pm i\sqrt{ab}}{a+b}=:\zeta_{\pm}$$
Integrating on a clockwise oriented dogbone contour, noting that as $z\to x\in(0,1)$ from above the real axis we find $f(z)$ tends to the integrand $(1-x)^\alpha x^\beta((1-x)^2a+x^2b)^{-\frac{\alpha+\beta}{2}}$ but as $z\to x\in(0,1)$ from below the real axis we find $f(z)$ tends to this quantity multiplied by $\exp(2\pi i\beta)\neq1$, we get: $$2\pi i(\res(f;\zeta_+)+\res(f;\zeta_-)+\res(f;\infty))=(1-\exp(2\pi i\beta))\cdot J$$Where $J$ is our desired integral.
The residue at $\zeta_\pm$ will equal the $(m-1)$th coefficient of the Taylor series of: $$z\mapsto\frac{\exp(\alpha\log_2(1-z))\exp(\beta\log_1(z))}{(a+b)^m(z-\zeta_{\mp})^m}=:g_{\pm}(z)$$When expanding about $\zeta_{\pm}$.
I'm not sure right now how to attack this in general, but I'll settle for the case $m=1$ now before I go to sleep.
Then it's easy: we just have to evaluate $g_{\pm}(\zeta_{\pm})$. We get: $$J=\frac{2\pi i}{1-\exp(2\pi i\beta)}(g_+(\zeta_+)+g_-(\zeta_-)+\res(f;\infty))$$
To identify $\res(f;\infty)$ we need to identify the $z^{-1}$ coefficient of the power series expansion of $f$ at $\infty$ - then multiply it by $(-1)$. In the knowledge that this series and the asymptotics are well-defined, to make life easy it suffices to consider the asymptotics as $z\to\infty$ along the positive real axis. With $z\gg0$ real we can compute: $$\begin{align}[z^{-1}]f(z)&=[z^{-1}]\frac{\exp(-\pi i\alpha)}{a+b}\frac{(1-z^{-1})^\alpha}{1-z^{-1}\cdot\frac{2a}{a+b}+z^{-2}\frac{a}{a+b}}\\&=[z^{-1}]\frac{\exp(-\pi i\alpha)}{a+b}(1-\alpha z^{-1}+o(z^{-1}))\left(1+z^{-1}\frac{2a}{a+b}+o(z^{-1})\right)\\&=\frac{\exp(-\pi i\alpha)}{a+b}\left(\frac{2a}{a+b}-\alpha\right)\end{align}$$Hence: $$\res(f;\infty)=\frac{\exp(-\pi i\alpha)}{a+b}\left(\alpha-\frac{2a}{a+b}\right)$$
Now for the ordinary residues.
The modulus of each $\zeta_{\pm}$ is $\sqrt{\frac{a}{a+b}}$. The $\log_1$-argument $\zeta_+$ is $\arctan\sqrt{\frac{b}{a}}=:\phi_1$ and the argument of $\zeta_-$ is $2\pi-\arctan\sqrt{\frac{b}{a}}$. The modulus of each $1-\zeta_{\pm}$ is $\sqrt{\frac{b}{a+b}}$ and the $\log_2$-arguments are $\mp\arctan\sqrt{\frac{a}{b}}=:\mp\phi_2$ respectively. So: $$\begin{align}g_+(\zeta_+)&=\frac{(b/(a+b))^{\alpha/2}\exp(-i\alpha\phi_2)(a/(a+b))^{\beta/2}\exp(i\beta\phi_1)}{2i\sqrt{ab}}\\&=\frac{b^{\alpha/2}a^{\beta/2}}{2i\cdot(a+b)\sqrt{ab}}\exp(i(\beta\phi_1-\alpha\phi_2))\end{align}$$And: $$g_-(\zeta_-)=-\frac{b^{\alpha/2} a^{\beta/2}}{2i\cdot(a+b)\sqrt{ab}}\exp(i(\alpha\phi_2-\beta\phi_1))\exp(2\pi i\beta)$$Concluding: $$J=-\frac{2\pi i\exp(-\pi i\alpha)}{(a+b)(1-\exp(2\pi i\beta))}\left(\frac{2a}{a+b}-\alpha\right)\\+\frac{\pi b^{\alpha/2}a^{\beta/2}}{(a+b)\sqrt{ab}}\left(\cos(\beta\phi_1-\alpha\phi_2)+i\frac{1+\exp(2\pi i\beta)}{1-\exp(2\pi i\beta)}\sin(\beta\phi_1-\alpha\phi_2)\right)$$
Which after some careful simplification becomes: $$\begin{align}\int_0^1\frac{(1-x)^\alpha\cdot x^\beta}{(1-x)^2a+x^2b}\d x&=\pi\cdot\frac{\beta a-\alpha b}{(a+b)^2}\csc(\pi\beta)\\+\pi\cdot\frac{b^{\alpha/2}a^{\beta/2}}{(a+b)\sqrt{ab}}&\cos\left(\alpha\arctan\sqrt{\frac{a}{b}}-\beta\arctan\sqrt{\frac{b}{a}}\right)\\+\pi\cdot\frac{b^{\alpha/2}a^{\beta/2}}{(a+b)\sqrt{ab}}&\cot(\pi\beta)\sin\left(\alpha\arctan\sqrt{\frac{a}{b}}-\beta\arctan\sqrt{\frac{b}{a}}\right)\\\int_0^1\frac{x^\beta\cdot(1-x)^{2-\beta}}{(1-x)^2a+x^2b}\d x&=\frac{\pi}{a+b}\csc\pi\beta\left(\beta-\frac{2b}{a+b}+\left(\frac{a}{b}\right)^{\frac{\beta-1}{2}}\sin\left(\frac{\pi\beta}{2}+2\arctan\sqrt{\frac{a}{b}}\right)\right)\end{align}$$For all positive real $a,b$ and real $-1<\alpha,\beta<3$ with $\alpha+\beta=2$ (regarding this as a limit for integer $\beta,\alpha$).
I have numerical evidence to suggest this revision of the post has no errors :)
I don’t have much hope for a closed form in general.