I am integrating the following integral
$$\int_{0}^\infty\frac{t^a}{1+t^2}dt$$ for $-1<a<0$.
by computing residues inside some contour. But I'm not sure what contour to use here, since $z^{a}$ gets large when $z$ is small, so I have to be careful about the part of the contour that gets near the origin.
Use the keyhole contour suggested, with outer radius $R$ and inner radius $\epsilon$. The contour integral becomes
$$\int_{\epsilon}^R dx \frac{x^a}{1+x^2} + i R \int_0^{2 \pi} d\theta \, e^{i \theta} \frac{R^a e^{i a \theta}}{1+R^2 e^{i 2 \theta}}\\ + e^{i 2 \pi a} \int_R^{\epsilon} dx \frac{x^a}{1+x^2} + i \epsilon \int_{2 \pi}^0 d\phi \, e^{i \phi} \frac{\epsilon^a e^{i a \phi}}{1+\epsilon^2 e^{i 2 \phi}}$$
Because $a \in (-1,0)$, the second and fourth integrals vanish as $R \to \infty$ and $\epsilon \to 0$, respectively.
On the other hand, the contour integral is also equal to $i 2 \pi$ times the sum of the residues at the poles $z=e^{i \pi/2}$ and $z=e^{i 3 \pi/2}$. Thus we have
$$\left ( 1-e^{i 2 \pi a}\right ) \int_0^{\infty} dx \frac{x^a}{1+x^2} = \frac{i 2 \pi}{2 i} \left (e^{i \pi a/2} - e^{i 3 \pi a/2} \right )$$
or
$$\int_0^{\infty} dx \frac{x^a}{1+x^2} = \frac{\pi}{2 \cos{(\pi a/2)}}$$