The problem I want to solve is:
Evaluate $$\int_{0}^{\infty} \frac{x^{\alpha}\log{x}}{1-x^2}dx,$$
where $0<\alpha<1$ using complex analysis.
I've learned several types of integrals involving $x^{\alpha}R(x)$ and $\log{x}R(x)$, where $R$ is a rational function. Also, I know how to solve certain integrals of the type $\int x^{\alpha}\log^{p}{x}R(x)dx$, using a method which is based on finding $\int x^{\alpha}R(x)dx$, and then differentiating with respect to $\alpha$. However, I can't use this method here, because $$\int_{0}^{\infty} \frac{x^{\alpha}}{1-x^2}dx$$ doesn't converge! I tried the following method: denote $f(z) = \frac{z^{\alpha}\log{z}}{1-z^2}$, and integrate $f(z)$ on the contour in the picture: 
Integrating $f$ over this contour, denote its boundary by $\partial D$, I get:
$$\int_{\partial D} f(z)dz = \int_{C_{R}} f(z)dz + (\int_{R}^{1+r} + \int_{1-r}^{\varepsilon}) \frac{e^{2\pi i \alpha}x^{\alpha}(\log{x}+2\pi i)}{1-x^2}dx + \int_{C_{r}} f(z)dz + \int_{C_{\varepsilon}} f(z)dz + (\int_{1+r}^{R} + \int_{\varepsilon}^{1-r}) \frac{x^{\alpha}\log{x}}{1-x^2}dx.$$
Here, $C_{R}$ is the large circle (minus a tiny part) in the picture, $C_{\varepsilon}$ is the tiny circle around $0$, and $C_{r}$ is the tiny circle around $1$ (minus tiny parts for both $C_{\varepsilon}$ and $C_{r}$). By Jordan's lemmas, the integral around $C_{R}$ goes to $0$ as $R \to +\infty$, as well as the integral around $C_{\varepsilon}$, while the integral around $C_{r}$ goes to $\pi^{2}e^{2 \pi i \alpha}$ as $r \to 0$. The $+2\pi i$ in the second integral is due to the branch of the logarithm, and the $e^{2 \pi i \alpha}$ is due to the branch of $z^{\alpha}$. Finally, I get:
$$ \int_{\partial D} f(z)dz = \int_{0}^{\infty} \frac{x^{\alpha}[(1-e^{2 \pi i \alpha})\log{x} - e^{2 \pi i \alpha}2 \pi i ]}{1-x^2}dx + \pi^{2}e^{2 \pi i \alpha}.$$
However, again, this integral does not converge.
Is there a way to repair this argument, or modify it a little bit, to get the solution?
EDIT: I forgot about the argument in the branch of $z^{\alpha}$, and due to that branch there's no need for $\log^{2}{z}$ in $f$, but I still can't use this method to find the integral for the same reason.

OK, you have done all of the contour integration correctly. Thus, all you really are missing is the fact that the "non-integrable" piece about which you are concerned is a Cauchy Principal Value, which is finite! So, let
$$I = \int_0^{\infty} dx \, \frac{x^{\alpha} \log{x}}{1-x^2} $$ $$J = PV \int_0^{\infty} dx \, \frac{x^{\alpha}}{1-x^2} $$
where the $PV$ denotes a Cauchy Principal Value. Now, the other piece you would need - and I assume you know how to do this - comes from the residue theorem. Because I assume you know what you are doing, I will skip this evaluation and just write the equation resulting from the application of the theorem:
$$\left ( 1-e^{i 2 \pi \alpha} \right ) I - i 2 \pi e^{i 2 \pi \alpha} J + \pi^2 e^{i 2 \pi \alpha} = \pi^2 e^{i \pi \alpha} $$
Now, we just need to equate real and imaginary parts and we get two equations in two unknowns $I$ and $J$:
$$(1-\cos{2 \pi \alpha}) I + 2 \pi \sin{ 2 \pi \alpha} \, J = \pi^2 (\cos{\pi \alpha} - \cos{ 2 \pi \alpha} )$$ $$\sin{2 \pi \alpha}\, I + 2 \pi \cos{ 2 \pi \alpha} \, J = \pi^2 (\sin{2 \pi \alpha} - \sin{ \pi \alpha} )$$
You can then eliminate $J$ and solve for $I$. The result is
$$I = \int_0^{\infty} dx \, \frac{x^{\alpha} \log{x}}{1-x^2} = -\pi^2 \frac{1-\cos{\pi \alpha}}{1-\cos{2 \pi \alpha}} = -\frac{\pi^2}{4 \cos^2{\left ( \frac{\pi}{2} \alpha \right )}}$$
You can also get $J$ if you want.