So a while ago I came across these types of integrals that has the form of $$ I = \int_a^b (x-a)^p(b-x)^{1-p}dx $$ Where $a$, $b$ and $p$ are real numbers, $a<b$ and $0<p<1$. And those types integral has a closed form, namely $$ I = \frac{\pi}{2}p(1-p)(b-a)^2\csc(p\pi) $$ But then I also came across integrals like these along the way $$ \int_0^3 \frac{x^{3/4}(3-x)^{1/4}}{5-x}dx $$ So I asked myself, is there a more generalized form of the $I$ integral, namely $$ J = \int_a^b \frac{(x-a)^p(b-x)^{1-p}}{\text{P}(x)} dx $$ Where $\text{P}(x)$ is any polynomial.
To begin with, I decided to make a more concise (?) definition of the function $$ \text{P}(x) = \prod_{k=1}^n (x-c_k)^{q_k} $$ Where $c_k$ is the $k$-th root of $\text{P}(x)$ and $q_k$ is the $k$-th root's multiplicity. $c_1 \not = c_2 \not = c_3 \not = \ldots \not = c_n$, $c_k \not \in (a, b)$, and if $a$ or $b$ is a root of $\text{P}(x)$ then they can't have a multiplicity of 2 or higher
First off, I'll analyze this function $$ f(z) = \frac{(z-a)^p (b-z)^{1-p}}{\text{P}(z)} \\ = \frac{|z-a|^p |b-z|^{1-p}}{\text{P}(z)} e^{i(p\arg(z-a) + (1-p)\arg(b-z))} $$ The poles of this function are $z = c_k$ with order $q_k$ for $k = 1,2,3,\ldots, n$
To begin, I integrated $f(z)$ over the dogbone contour

Which means I'll have $$ \oint_{\text{Dogbone}} f(z) dz = \left(\int_{\gamma_1} + \int_{\gamma_2} + \int_{\mu_1} + \int_{\mu_2}\right) f(z) dz $$ If we call the radius of the $\mu$ paths $\varepsilon$, and have it approach 0, then both integrals over $\mu_1$ and $\mu_2$ will go to 0 also. The integral over $\gamma_1$ will be $J$ and the integral over $\gamma_2$ will be $-e^{(1-p)2\pi} J$.
Which means
$$
\oint_{\text{Dogbone}} f(z) dz = \left(1 - e^{(1-p)2\pi}\right) J
$$
Now, to continue, we can inflate the contour into this

I'll call the big circular contour $\Gamma$ and the small ones surrounding the poles at $c_k$ $\omega_k$. Which means we'll have $$ \left(1 - e^{(1-p)2\pi}\right) J = \left(\oint_\Gamma + \sum_{k=1}^n \oint_{\omega_k}\right) f(z) dz $$ For the $\omega$ integrals, we can simply use residue theorem for higher order poles $$ \oint_{\omega_k} f(z) dz = - \frac{2\pi i}{(q_k - 1)!} \lim_{z \rightarrow c_k} \partial_z^{q_k - 1} (z-c_k)^{q_k} f(z) $$ Plugging it back into the equation $$ \left(1 - e^{(1-p)2\pi}\right) J = \oint_\Gamma f(z) dz - \sum_{k=1}^n \frac{2 \pi i}{(q_k - 1)!} \lim_{z \rightarrow c_k} \partial_z^{q_k - 1} (z-c_k)^{q_k} f(z) $$
Now, my main problem is the final integral. I know that it'll be the residue of $f(z)$ at infinity, I know there's a formula for it, but I have no idea of how to evaluate such residues. My guess is that it'll some how evaluate to 0, but still, I have no idea how that'll be the answer.
26/2 Edit: Fixed some of the mistakes in calculation + Thanks to @Diger for pointing out the mistake in the inflated contour
$\newcommand{\d}{\,\mathrm{d}}\newcommand{\res}{\operatorname{Res}}$Define $z\mapsto z^p$ via the principal logarithm $(-\pi\le\arg z<\pi)$ and $z\mapsto z^{1-p}$ via the logarithm induced by $0\le\arg<2\pi$. Let's call these two argument functions $\arg_1,\arg_2$ respectively.
Let's take a polynomial $\lambda$ which has no roots on $[a,b]$ (no assumptions other than that). $p$ is in $(0,1)$ and it is important that $b>a\ge0$. The case $a=0$ requires special attention in the final case for linear $\lambda$.
Then: $f:\Bbb C\to\hat{\Bbb C}$, $z\mapsto(z-a)^p(b-z)^{1-p}\lambda(z)^{-1}$ is meromorphic outside of $[a,b]$. This depends in an essential way on $\arg_1(-x)=-\pi$ when $x$ is a positive real and on the combination $p\arg_1+(1-p)\arg_2$.
You want to know the value of $\res(f,\infty)$. Assuming the rest of your work is correct, knowledge of this will give you the value of your integral.
Let $R$ be a real number larger than the modulus of every root of $\lambda$.
If $|z|\ge R$ then $|f|=\frac{|z|}{|\lambda(z)|}\cdot|1-a/z|^p\cdot|1-b/z|^{1-p}$. If $\lambda$ has degree greater than two, then: $$|z\cdot f(z)|\overset{|z|\to\infty}{\longrightarrow}0$$
Which is a sufficient condition for $\res(f,\infty)=0$, so you get to ignore this term
More generally, if $f$ is holomorphic on some large annulus and vanishes as $|z|\to\infty$ then $-\lim_{|z|\to\infty}z\cdot f(z)=\res(f,\infty)$. This is justified by the "series expansion at infinity". We can apply this when the degree of $\lambda$ is two: say, $\lambda(z)=\alpha\cdot z^2+\beta\cdot z+\gamma$ where $\alpha\neq0$.
Then: $$\begin{align}\res(f,\infty)&=-\lim_{|z|\to\infty}\frac{z\cdot |z|}{\alpha\cdot z^2+\beta\cdot z+\gamma}|1-a/z|^p|1-b/z|^{1-p}e^{ip\arg_1(z-a)+i(1-p)\arg_2(b-z)}\\&=-\lim_{|z|\to\infty}\frac{1}{\alpha\cdot\frac{z}{|z|}}e^{ip\arg_1(z-a)+i(1-p)\arg_2(b-z)}\end{align}$$
If $z$ is on the negative real axis, the expression is $\frac{1}{-\alpha}e^{-i\pi p}$. This is cheating really, but since the limit must exist, it in particular must equal the value as $|z|\to\infty$ in the negative real axis so we can conclude: $$\res(f,\infty)=\frac{e^{-\pi ip}}{\alpha}$$
Finally, say $\lambda(z)=\alpha\cdot z+\beta$ is linear, where $\alpha\neq0$. Then we need a different approach, as $f$ no longer vanishes as $|z|\to\infty$. The definition $\res(f,\infty)=-\res(f(1/z)/z^2,0)$ becomes useful now.
There are some subtle aspects in the evaluation below. I casually pretend the things we are used to from real analysis apply for complex numbers, but this is not so. I mention the subtleties at the end.
If $\gamma:=\beta/\alpha$, then: $$\frac{1}{z^2}f(1/z)=\frac{e^{\pi i(1-p)}}{\alpha\cdot z^2}\cdot\frac{1}{1+\gamma z}(1-az)^p(1-bz)^{1-p}$$
Expand in series: $$\frac{1}{z^2}f(1/z)=-\frac{e^{-i\pi p}}{\alpha\cdot z^2}\cdot(1-\gamma z+o(z))(1-paz+o(z))(1-(1-p)bz+o(z))$$
We want to extract the $z^{-1}$ coefficient: $$-\frac{e^{-i\pi p}}{\alpha}(-\gamma-pa-(1-p)b)$$
Concluding: $$\res(f,\infty)=\frac{e^{-i\pi p}}{\alpha}\cdot(p(b-a)-b-\gamma)$$
This last expression, for example, gives the correct value of the integral for the example on Wikipedia (which is much less carefully done, I might add).
We can witness the formula for the quadratic case in action for the following situation (exercise):
NOTE: I used the equalities $(1-az)^p/z^p=((1/z)-a)^p$, $(bz-1)^{1-p}/z^{1-p}=(b-(1/z))^{1-p}$ and $(1/z^p)\cdot(1/z^{1-p})=1/z$. These are actually quite subtle: so subtle, that they are not even always correct. However, I leave it as an exercise in geometric reasoning to show that the product of all these ‘equations’ is a genuine equation for small $z$.
Expanding in binomial series is also subtle. I presented it in the simplest, but technically misleading, way. So long as $|z|<1/a$, $(1-az)^p$ is holomorphic and we can do the usual Taylor expansion. But for small $z$, $(1-bz)^{1-p}$ is not holomorphic. But, $(bz-1)^{1-p}$ is holomorphic for small $z$ and we get the usual Taylor series multiplied by a factor of $(-1)^{1-p}=-e^{-i\pi p}$.
Phew!