I am trying to compute
$$\int_0^\infty \frac{\ln^2 x}{1+x^2} dx $$
using a complex contour integration of $f(z) = \ln^2 z / (1+z^2)$ around a closed contour with the segments:
$$C_1 : [-R,-\rho]\\C_2: \{z: |z| = \rho, 0 \leq \arg z \leq \pi\}\\ C_3: [\rho , R] \\ C_4: \{z: |z| = R, 0 \leq \arg z \leq \pi\}.$$
This contour encloses one singular point $z = i$ where I found $Res(f,i) = i\pi^2/8.$
My difficulty is in handling the $\int_{C_1}f(z) dz$ integral on the real interval $[-R,-\rho]$ which does not vanish as $R \to \infty$ and $\rho \to 0$.
Thank you.
By the residue theorem
$$\oint_{C_1 \cup C_2 \cup C_3 \cup C_4}f(z) \, dz = 2\pi i Res(f,i) = 2\pi i (i \pi^2/8) = -\frac{\pi^3}{4}.$$
It is straightforward to show that the integrals over $C_2$ and $C_4$ vanish in the limit as $R \to \infty$ and $\rho \to 0$.
Hence,
$$\tag{1} -\frac{\pi^3}{4} = \int_0^\infty \frac{\ln^2 x}{1 + x^2} \, dx + \lim_{R \to \infty, \rho \to 0} \int_{C_1} \frac{\ln^2 z}{1 + z^2} \, dz.$$
Now we can focus on the second integral on the RHS -- the source of your difficulty.
Note that
$$\begin{align}\lim_{R \to \infty, \rho \to 0} \int_{C_1} \frac{\ln^2 z}{1 + z^2} \, dz &= \int_{-\infty}^0 \frac{ \ln^2 x}{1 + x^2} \, dx \\ &= \int_{0}^\infty \frac{ \ln^2 (-x)}{1 + x^2} \, dx \\ &= \int_{0}^\infty \frac{ (\ln x + i \pi)^2}{1 + x^2} \, dx \\ &= \int_{0}^\infty \frac{ \ln^2 x }{1 + x^2} \, dx + 2\pi i\int_{0}^\infty \frac{ \ln x }{1 + x^2} \, dx - \pi^2\int_{0}^\infty \frac{ 1 }{1 + x^2} \, dx \\ &= \int_{0}^\infty \frac{ \ln^2 x }{1 + x^2} \, dx + 0 - \pi^2 \frac{\pi}{2} \\ &= \int_{0}^\infty \frac{ \ln^2 x }{1 + x^2} \, dx - \frac{\pi^3}{2}\end{align}.$$
The second of the three integrals on the RHS can be shown to be $0$ by showing that integrals over $[0,1]$ and $[1,\infty)$ must cancel (using a change of variables $u = 1/x)$.
Substituting into (1) we obtain
$$-\frac{\pi^3}{4} = 2\int_0^\infty \frac{\ln^2 x}{1 + x^2} \, dx - \frac{\pi^3}{2}.$$
Thus,
$$\int_0^\infty \frac{\ln^2 x}{1 + x^2} \, dx = \frac{\pi^3}{8}.$$