Show $$\int^{\infty}_{0}\frac{\log(x)}{x^a(x+1)}dx = \frac{\pi^2\cos(\pi a)}{\sin^2(\pi a)}, \ 0<a<1$$
I have tried tackling this by using a keyhole domain, where $\gamma_{\epsilon}$, where $\epsilon$ is small, is a circle around the origin, and $\Gamma_{R}$ is the circle of radius R around the origin, where R is large. My branch cut for log(z) is from $0<\theta<2\pi$ for $z=re^{i\theta}$. So far I have: $$\int_{\delta D}\frac{\log(z)}{z^a(z+1)}dz=\int^R_{\epsilon}\frac{\log(x)}{x^a(x+1)} dz+\int_{\Gamma_R} \frac{\log(z)}{x^a(z+1)}dz+\int_R^{\epsilon} \frac{\log(x)}{e^{i2\pi a}x^a(x+1)}dx+\int_{\gamma_\epsilon} \frac{\log(z)}{z^a(z+1)}dz$$ Using ML-estimates, I can show the two gamma integrals go to zero as epsilon and R go to zero and infinity. Also, by the residue theorem, $$\int_{\delta D}\frac{\log(z)}{z^a(z+1)}dz=2\pi i\sum Res[\frac{\log(z)}{z^a(z+1)},-1]=2\pi i\frac{i\pi}{e^{i\pi a}}=-\frac {2\pi^2}{e^{i\pi a}}$$ Putting this together with the previous equation we get: $$\int^R_{\epsilon}\frac{\log(x)}{x^a(x+1)} dz=-\frac {2\pi^2}{e^{i\pi a}}(1-\frac{1}{e^{i2\pi a}})$$
Now I am lost on how to show the above equation. Any hints?
I would first evaluate $$\int_{0}^{\infty} \frac{1}{x^{a}(1+x)} \, dx$$
and then differentiate under the integral sign.
Let $ \displaystyle f(z) = \frac{1}{z^{a}(1+z)}$ and integrate around the keyhole contour that you described.
Then
$$ \int_{0}^{\infty} \frac{1}{x^{a}(1+x)} \, dx + \int_{\infty}^{0} \frac{1}{x^{a}e^{2 \pi i a} (1+x)} \, dx = 2 \pi i \ \text{Res}[f(z),-1] = \frac{2 \pi i}{e^{\pi i a}} \, ,$$
which means
$$ \int_{0}^{\infty} \frac{1}{x^{a}(1+x)} \, dx = 2 \pi i \ \frac{e^{- \pi i a}}{1-e^{-2 \pi i a}}= \pi \, \frac{2i}{e^{\pi ia} - e^{- \pi i a}} = \pi \csc (\pi a).$$
Then differentiating under the integral sign we get
$$ \begin{align} - \int_{0}^{\infty} \frac{\ln x}{x^{a}(1+x)} \, dx &= \pi \frac{d}{da} \csc (\pi a) \\ &= \pi \Big( - \pi \cot (\pi a) \csc(\pi a) \Big) \\ &= - \frac{ \pi^{2}\cos (\pi a)}{\sin^{2} (\pi a)} \end{align}$$
and the results follows.
EDIT:
I'm also going to complete the evaluation you started.
Let $ \displaystyle f(z) = \frac{\log z}{z^{a}(1+z)}$ and integrate around the same contour.
Then
$$ \int_{0}^{\infty} \frac{\log x}{x^{a}(1+x)} \, dx + \int_{\infty}^{0} \frac{\log x + 2 \pi i}{x^{a}e^{2 \pi i a}(x+1)} \, dx = 2 \pi i \text{Res} [f(z), -1] = -2 \pi^{2} e^{- \pi i a}.$$
Rearranging, we have
$$ (1- e^{- 2 \pi i a}) \int_{0}^{\infty} \frac{\log x}{x^{a}(1+x)} \, dx - 2 \pi i \int_{0}^{\infty} \frac{1}{x^{a}(1+x)} \, dx = - 2 \pi^{2}e^{- \pi i a}.$$
And multiplying both sides of the equation by $e^{\pi i a}$, we get
$$ 2i \sin (\pi a) \int_{0}^{\infty} \frac{\log x}{x^{a}(1+x)} \, dx - 2 \pi i e^{ -\pi i a} \int_{0}^{\infty} \frac{1}{x^{a}(x+1)} \, dx = - 2 \pi^{2}.$$
Call the first integral $I$ and the second integral $J$.
Equating the real and imaginary parts on both sides of the equation, we obtain the system of equations
$$ - 2 \pi \sin(\pi a) J = - 2 \pi^{2} $$
$$ 2 \sin(\pi a) I - 2 \pi \cos(\pi a) J = 0.$$
Solving the system for $I$, we obtain
$$ I = \int_{0}^{\infty} \frac{\log x}{x^{a}(1+x)} \ dx = \frac{2 \pi \cos(\pi a) \left(\frac{\pi}{\sin (\pi a)} \right)}{2 \sin (\pi a)} = \frac{\pi^{2} \cos (\pi a)}{\sin^{2}(\pi a)}.$$