I'm asked to show the following equality given $a\in (-1,1)\subset\Bbb R$
$$\int\limits_0^\infty\frac{x^a\ \log(x)}{(1+x)^2}dx=\frac{\pi\sin(\pi a)-a\pi^2\cos(\pi a)}{\sin^2(a\pi)}$$
So I'm trying to use the keyhole contour (as shown here) and so far I've been able to see the integrals along the circunferences tend to $0$ as $R\to\infty$ and $\varepsilon\to 0$. Computing $\text{Res}[f(z);-1]$ (where $f(z)$ equals the integrand) I get
$$\lim_{z\to -1}\frac{d}{dz}(1+z)^2f(z)=\lim_{z\to -1}az^{a-1}\log(z)+z^{a-1}=e^{\pi i(a-1)}(a(1+\pi i)+1)$$
On the other side,
$$\begin{align*}\int_\varepsilon^Rf(z)dz+\int_R^\varepsilon f(z)dz&=\int_\varepsilon^R\frac{z^a\log|z|}{(1+z)^2}dz+\int_R^\varepsilon\frac{z^a(\log|z|+2\pi i)}{(1+z)^2}dz\\ &=\int_\varepsilon^R\frac{z^a\log|z|}{(1+z)^2}dz+\int_R^\varepsilon\frac{z^a\log|z|}{(1+z)^2}dz-2\pi i\int_\varepsilon^R\frac{z^a}{(1+z)^2}\end{align*}$$
I know the result of the last integral, but I'm not sure whether what I've done is right, and what to do to finish it. Any help is appreciated.
The point of the keyhole contour is to exploit the multivaluedness of the function being integrated. In this case, the function is $z^a \log{z}$. The right-hand side should look like
$$\int_0^{\infty} dx \frac{x^a \log{x}}{(1+x)^2} - e^{i 2 \pi a} \int_0^{\infty} dx \frac{x^a (\log{x}+i 2 \pi)}{(1+x)^2}$$
So you were halfway there - you had the $i 2 \pi$ from the log, but you also needed the other factor from the $z^a$.
The integral over the circular arcs vanish in the limits of $R \rightarrow \infty$ and $\epsilon \rightarrow 0$. It appears you are attempting to show this. I will leave the rest of the details to the reader.
Now, I leave it to the reader to show that
$$\int_0^{\infty} dx \frac{x^a}{(1+x)^2} = \frac{\pi a}{\sin{\pi a}}$$
(which result you claim to have), so that
$$\left (1-e^{i 2 \pi a}\right) \int_0^{\infty} dx \frac{x^a \log{x}}{(1+x)^2} - i 2 \pi e^{i 2 \pi a} \frac{\pi a}{\sin{\pi a}} = i 2 \pi (-1) e^{i \pi a} (1+i \pi a)$$
(I get $1+i \pi a$, not $1+(1+i \pi)a$, in the residue.)
Do out the algebra - the sought result follows.