Could someone have any idea on how to solve the above integral. On my last attempt I did the following:
Change of variable $u=x^b \Rightarrow du=bx^{(b-1)}dx$
The integral turned into:
$\frac{1}{b}\int_{0}^\infty\frac{1}{u^{\frac{b-a}{b}}(1-u)}dx$
Extended this integral to the complex plane. This integral has two poles on $u=0$ and $u=1$
Calculate the path integral over the closed contour $C$ (clockwise direction) composed of large $1/4$ of circle on the first quadrant ($C_R$) with radius $R$, and centered @ $0$, connected to two lines on the real and imaginary axis both with one end on @ $0$. The final path would be similar to a $1/4$ pizza shape. The poles would be contored by small arcs of cirles of radius $\epsilon$ around them. (Perhaps a drawing would be more appropriate :( )
The contour integral would be:
$\oint_C=\int_{C_R}+PV\int^\infty_{0}+\int_\text{left small 1/4 circle centered @ (0,0)}+\int_\text{right small semicircle centered @ (1,0)}+\int^{Ri}_{\epsilon i}$
$\epsilon \to 0$
$R \to \infty$
By the residue theorem $\oint_C$ is zero because there are no poles inside it.
$\int_{C_R}$ is zero as $R \to \infty$.
I tried to calculate the other remaining integrals but I did not get the correct answer which is
$\frac{\pi}{b}cot\frac{\pi a}{b}$
Thanks in advance.
Too long for a comment
I'm afraid it would be difficult to evaluate the integral in the complex plane using a quarter of a big circle - because in this case you get another integral $\int_{Ri}^{\epsilon i}$, which should be evaluated as well.
The basic idea of the complex integration is to use available symmetries. In your case I would recommend a keyhole contour with two small half-circles (clockwise direction) around $x=1$ on upper and lower banks of the cut (the cut is from $0$ to $\infty$).
Using this contour we can evaluate integral type $I(d)=\int_{0}^\infty\frac{1}{u^d(1-u)}dx$, where $d\in(0,1)$
$$\oint=I(d)+\int_{C_1}+\int_R\,\,-e^{-2\pi id}I(d)+\int_{C_2}+\int_r =2\pi i\sum Res$$ where $\int_{C_1}$ and $\int_{C_2}$ are integrals around $x=1$ on the upper and lower bank of the cut correspondingly.
It can be evaluated that integral along a big circle $\to0$ as $R\to\infty$, and integral around $x=0$ (small circle) $\to0$ as $r\to0$ (at given $d$).
Because there are no poles inside the closed contour, we get $$I(d)\bigl(1-e^{-2\pi id}\bigr)=-\bigl(\int_{C_1}+\int_{C_2}\bigr)$$ where integral $\int_{C_2}$ has the additional phase (multiplier) $e^{-2\pi id}$.
Could you proceed from here?