I solved this question and had a try as shown ... i got similar form ... but not getting correct answer. here is the question:
Question 3.
$\quad$ (a) By considering the integral of $z^{-1/2}\log^2(z)/(1+z^2)$ around a suitably chosen contour in the cut $z$ plane, prove that $$\int_0^{\infty}\frac{\log^2(t) - \pi^2}{t^{1/2}(1+t^2)}dt = -\frac{\pi^3}{4\sqrt{2}}$$
$\quad$ In your solution, include a diagram that clearly specifies the contour you are using and provide a careful discussion of all required estimates of contributions from contours that you will discard when an appropriate limit is taken.
$\quad$ (b) Using the same integration contour as in part (a) with the integrand $z^{-1/2}\log^2(z)/(1-z)$, find a relation between the integrals $$\int_0^{\infty}\frac{\log^2(t)}{t^{1/2}(1+t)}dt \quad\text{ and }\quad \int_0^{\infty}\frac{dt}{t^{1/2}(1+t)}.$$ $\quad$Use this relation to deduce, without further contour integration, that $$\int_0^{1}\frac{\log^2(t)}{t^{1/2}(1+t)}dt = \frac{\pi^3}{2}.$$
Here is what I have so far:
$\quad$ Ans 3. a) $$\int_0^{\infty}\frac{log^2(t) - \pi^2}{\sqrt{t}(1+t^2)}dt$$
$$\begin{multline}
\shoveleft \text{Consider} \quad f(z) = \frac{log^2(z)}{\sqrt{z}(1+z^2)} \\
\shoveleft \text{Consider} \quad \int_Cf(z)dz \qquad \text{along the C as shown below}
\end{multline}$$
$$\begin{multline} \shoveleft \text{Poles are at } z = 0 \text{ (not lies[sic] in contour)}, \quad z = \pm i \\ \shoveleft \text{Res }f(z=i) \quad \space \space \space = \quad \lim_{z \rightarrow i}\frac{(z-i)log^2(z)}{\sqrt{z}(z-i)(z+i)} \space = e^{i\pi /4}\frac{\pi^2}{8} \\ \shoveleft \text{Res }f(z=-i) \quad = \quad \lim_{z \rightarrow -i}\frac{(z+i)log^2(z)}{\sqrt{z}(z-i)(z+i)} = -e^{3i\pi /4}\frac{\pi^2}{8} \\ \shoveleft \therefore \quad \int_{C}f(z)dz = 2\pi i\left[\frac{\pi^2}{8}(e^{i\pi /4} - e^{3i\pi /4}) \right] = \frac{i\pi^3}{2\sqrt{2}} \\ \shoveleft \\ \shoveleft \text{Thus,} \\ \shoveleft \quad \int_{AB}f(z)dz + \int_{\Gamma}f(z)dz + \int_{FG}f(z)dz + \int_{\gamma}f(z)dz = \frac{i\pi^3}{2\sqrt{2}} \\ \shoveleft \\ \shoveleft \text{On AB: } z = x \quad \text{and on FG: } z = xe^{2\pi i} \\ \shoveleft \therefore \quad \int_{AB}f(z)dz + \int_{FG}f(z)dz = \int_{r}^{R}\frac{log^2x}{\sqrt{x}(1+x^2)}dx + \int_{R}^{r}\frac{\left( log(xe^{2\pi i}) \right)^2e^{2\pi i}}{\sqrt{xe^{2\pi i}}(1+x^2e^{2\pi i})}dx \\ \shoveleft \\ \shoveleft = \int_{r}^{R}\frac{log^2x}{\sqrt{x}(1+x^2)}dx + \int_{R}^{r}\frac{log^2x - 4\pi^2 +4\pi log(x)i}{\sqrt{x}e^{\pi i}(1+x^2)}dx \\ \shoveleft \\ \shoveleft = \int_{r}^{R}\frac{log^2x}{\sqrt{x}(1+x^2)}dx \ + \space \int_{r}^{R}\frac{log^2x - 4\pi^2 + 4\pi log(x)i}{\sqrt{x}(1+x^2)}dx \quad \because \space e^{\pi i} = -1 \\ \shoveleft \\ \shoveleft = \int_{r}^{R}\frac{2log^2x - 4\pi^2 - 4\pi log(x)i}{\sqrt{x}(1+x^2)}dx \\ \end{multline}$$
$$\begin{multline} \shoveleft \\ \shoveleft \text{For the circle } \gamma : \space z = re^{i \theta} \\ \shoveleft \text{So, } \int_{\gamma}f(z)dz = \int_{2\pi}^{0} \frac{\left( log(re^{i \theta})\right)^2(rie^{i \theta})}{\sqrt{re^{i \theta}}(1 + r^2 e^{2i \theta})}d\theta = \int_{2\pi}^{0}\frac{\sqrt{r}*ie^{i \theta}*log^2(re^{i \theta})}{e^{i\theta /2}(1 + r^2e^{2i\theta})}d\theta \\ \shoveleft \text{Clearly, if } r \rightarrow 0, \space \int_{\gamma}f(z)dz \rightarrow 0 \\ \end{multline}$$
$$\begin{multline} \shoveleft \\ \shoveleft \text{Now, on the circle } \Gamma : \space z = Re^{i\theta} \\ \shoveleft \text{So, } \int_{\Gamma}f(z)dz = \int_{0}^{2\pi} \frac{\left( log(Re^{i \theta})\right)^2(Rie^{i \theta})}{\sqrt{Re^{i \theta}}(1 + R^2 e^{2i \theta})}d\theta = \int_{0}^{2\pi}\frac{\sqrt{R}*ie^{i \theta /2}*log^2(Re^{i \theta})}{e^{i\theta /2}(1 + R^2e^{2i\theta})}d\theta \\ \shoveleft = \int_{0}^{2\pi}\frac{ie^{i\theta /2}*log^2(Re^{i\theta})}{R^{3/2}\left(\frac{1}{R^2} + e^{2i\theta}\right)}d\theta \\ \shoveleft \text{if } R \rightarrow \infty, \space \int_{\Gamma}f(z)dz \rightarrow 0 \\ \shoveleft \\ \shoveleft \text{So, we have } \\ \shoveleft \quad \int_{0}^{\infty}\frac{2log^2x - 4\pi^2 +4\pi log(x)i}{\sqrt{x}(1+x^2)}dx = \frac{i\pi^3}{2\sqrt{2}} \\ \shoveleft \quad \qquad \implies +\int_{0}^{\infty}\frac{4\pi log(x)}{\sqrt{x}(1+x^2)}dx = \frac{\pi^3}{2\sqrt{2}} \\ \shoveleft \space \space \qquad \quad \qquad \text{or } \int_{0}^{\infty}\frac{log(x)}{\sqrt{x}(1+x^2)}dx = \frac{\pi^2}{2\sqrt{2}} \end{multline}$$
I don't understand the explanation in the solution, can anyone explain where is my mistake ? How to arrive at the answer?
You chose the branches of $\sqrt z$ and $\ln z$ corresponding to $\arg z \in [0, 2 \pi)$. Then $\sqrt {-i} = e^{3 \pi i/4}$, $\ln(-i) = 3 \pi i/2$, and, fixing the typo in the $\ln x$ term, you get $$\int_0^\infty \frac {2 \ln^2 x + 4 \pi i \ln x - 4 \pi^2} {\sqrt x \, (1 + x^2)} dx = 2 \pi i \left( \frac {\ln^2 i} {\sqrt i} \operatorname*{Res}_{x = i} \frac 1 {1 + x^2} + \frac {\ln^2(-i)} {\sqrt {-i}} \operatorname*{Res}_{x = -i} \frac 1 {1 + x^2} \right) = \\ -\frac {\left( \frac 5 2 + 2 i \right) \pi^3} {\sqrt 2}.$$ Just take the principal branches and choose a keyhole contour avoiding $(-\infty, 0]$. You'll get $$\int_{-\infty}^0 \frac {2 \ln^2 |x| - 2 \pi^2} {i \sqrt{|x|} \, (1 + x^2)} dx = 2 \pi i \left( \operatorname*{Res}_{x = -i} + \operatorname*{Res}_{x = i} \right) \frac {\ln^2 x} {\sqrt x \, (1 + x^2)},$$ which is exactly what is required for a).