$${\dfrac{1}{\pi^2}}\int_{0}^{\infty}\dfrac{{(\ln{x}})^2}{\sqrt{x}{(1-x)^2}} \mathrm d x$$
I tried substituting $1/x$ for $x$ but the the only change in the integral is that the $\sqrt{x}$ moves in the numerator from the denominator. I don't understand what to substitute. $\tan{x}$ doesn't seem to work.
On
Letting $t=\sqrt x$ transforms the integral into $$ I=8 \underbrace{\int_0^{\infty} \frac{\ln ^2 t}{\left(1-t^2\right)^2} d t}_J $$ For $J$, $t\mapsto \frac 1t$ gives another version of $J$
$$ J=\int_0^{\infty} \frac{t^2 \ln ^2 t}{\left(1-t^2\right)^2} d t $$ Summing two versions of $J$ yields $$ \begin{aligned} 2 J&=\int_0^{\infty} \frac{1+t^2}{\left(1-t^2\right)^2} \ln ^2 t d t \\ & =\int_0^{\infty} \frac{1+\frac{1}{t^2}}{\left(t-\frac{1}{t}\right)^2} \ln ^2 t d t \\ & =\int_0^{\infty} \ln ^2 t \frac{d\left(t-\frac{1}{t}\right)}{\left(t-\frac{1}{t}\right)^2} \\ & =-\int_0^{\infty} \ln ^2 t d\left(\frac{1}{t-\frac{1}{t}}\right) \\ & =2 \int_0^{\infty} \frac{\ln t}{t^2-1} d t \\ & =2\left(\frac{\pi^2}{4}\right) \\ & \end{aligned} $$
We can now conclude that $$\boxed{\frac{I}{\pi^2} =2}$$
You can split the integral at $1$ and use your substitution in the second part. Then the integral becomes $$ I \equiv \int \limits_0^\infty \frac{\ln^2 (x)}{\sqrt{x} (1-x)^2} \, \mathrm{d} x= \int \limits_0^1 \left(\frac{1}{\sqrt{x}} + \sqrt{x}\right) \frac{\ln^2 (x)}{(1-x)^2} \, \mathrm{d} x \, .$$ Now let $x = \mathrm{e}^{-2 t} $ and integrate by parts to obtain $$ I = 4 \int \limits_0^\infty \frac{t^2 \cosh(t)}{\sinh^2(t)} \, \mathrm{d} t = 8 \int \limits_0^\infty \frac{t}{\sinh(t)} \, \mathrm{d} t = 16 \int \limits_0^\infty \frac{t \mathrm{e}^{-t}}{1 - \mathrm{e}^{-2 t}} \, \mathrm{d} t\, .$$ Finally, use the geometric series to expand the denominator and recognise the Dirichlet lambda function: $$ I = 16 \sum \limits_{n=0}^\infty \int \limits_0^\infty t \mathrm{e}^{-(2n+1) t} \, \mathrm{d} t = 16 \sum \limits_{n=0}^\infty \frac{1}{(2n+1)^2} = 16 \lambda (2) = 12 \zeta(2) = 2 \pi^2 \, .$$ Therefore, $\frac{1}{\pi^2} I = 2$ .