I was trying next: $I=\int_0^\infty \frac {\ln^2(x)\ln(1+ax)}{1+x^2}$; $\frac{dI}{da}=\int_0^\infty \frac{ x\ln^2(x)}{(1+x^2)(1+a*x)}\ dx$;
$f(z)= \frac {z\ln^3(z)}{(1+az)(1+z^2)}$ ; $f(\overline{z})= \frac {z(\ln(z+2\pi*i))^3}{(1+az)(1+z^2)}$;
$\textrm{Res}_{z=i}f(z)=-\frac{\pi^3(i+a)}{16(a^2+1)}$; $\textrm{Res}_{z=-i}f(z)=\frac{\pi^3(i-a)}{16(a^2+1)}$; $\textrm{Res}_{z=-\frac{1}{a}}f(z)=\frac{(\ln(a)-\pi i)^3}{a^2+1}$;
sum of integrals around the contour: $\int_0^\infty\frac {z\ln^3(z)}{(1+az)(1+z^2)}dz-(\int_0^\infty\frac {z\ln^3(z)}{(1+az)(1+z^2)}dz+6\pi i \int_0^\infty\frac {z\ln^2(z)}{(1+az)(1+z^2)}dz-12\pi^2\int_0^\infty\frac {z\ln(z)}{(1+az)(1+z^2)}dz-8\pi^3 i\int_0^\infty\frac {z}{(1+az)(1+z^2)}dz)$ $\int_0^\infty\frac {z}{(1+az)(1+z^2)}dz=\frac{\pi a-2\ln((a)}{2(a^2+1)}$,
so, $-6\pi i\frac{dI}{da}+8\pi^3 i\frac{\pi a-2\ln((a)}{2(a^2+1)}=2\pi i\sum{Res}=\frac{8\pi i \ln^3(a)-24\pi i\ln^2(a)-24\pi^2\ln(a)-8\pi^3 i-\pi^3 a}{4(1+a^2)} $;
taking imaginary parts:
$\frac{dI}{da}=\frac{17\pi^3 a-8\pi^2\ln(a)-8\ln^3(a)}{24(a^2+1)}$, and $I=\int_{0}^{1}\frac{17\pi^3 a-8\pi^2\ln(a)-8\ln^3(a)}{24(a^2+1)}da$,
but final answer doesn't connect with mathematica. Where is my mistake?

To apply Feynman's trick is a nice idea:
$$ I=\int_{0}^{+\infty}\frac{\log^2(x)\log(1+x)}{1+x^2}\,dx = \int_{0}^{1}\int_{0}^{+\infty}\frac{x\log^2(x)}{(1+x^2)(1+a x)}\,dx\,da \tag{1}$$ leads to: $$ I = \frac{\pi^3}{8}\int_{0}^{1}\frac{a\,da}{1+a^2}-\frac{\pi^2}{3}\int_{0}^{1}\frac{\log(a)}{1+a^2}\,da-\frac{1}{3}\int_{0}^{1}\frac{\log^3(a)}{1+a^2}\,da \tag{2} $$ and since $\int_{0}^{1}a^k\log(a)\,da = -\frac{1}{(k+1)^2}$, $\int_{0}^{1}a^k\log^3(a)\,da = -\frac{6}{(k+1)^4}$ it turns out that
$$\boxed{ I = \color{red}{\frac{\pi^3\log 2}{16}+\frac{\pi^2}{3}\sum_{k\geq 0}\frac{(-1)^k}{(2k+1)^2}+2\sum_{k\geq 0}\frac{(-1)^k}{(2k+1)^4}}}\tag{3} $$ depending on quite non-elementary constants, as already pointed out by Zaid Alyafeai.