I would like to evaluate this integral,$$\int_{0}^{1}\frac{\arctan\left(\frac{ax}{1-x}\right)}{\sqrt{1-x}}\frac{dx}{x^{3/2}}\tag1$$
This is the approach I will take:
We can begin with a sub: $y=\sqrt{x}$, $dx=2\sqrt{x}dy$
$$-2\int_{0}^{1}\frac{\arctan\left(\frac{ay^2}{y^2-1}\right)}{\sqrt{1-y^2}}\frac{dy}{y^2}\tag2$$
Not sure, but we can try integration by parts: $u=\arctan\left(\frac{ay^2}{y^2-1}\right)$, $$du=\frac{2ay}{y^2-1}-\frac{2ay^3}{(y^2-1)^2}\times \frac{1}{\frac{a^2y^4}{(y^2-1)^2}+1}dy$$
$dv=\frac{1}{y^2\sqrt{1-y^2}}dy$, $$v=-\frac{\sqrt{1-y^2}}{y}$$
$$2a\int_{0}^{1}\frac{(1-y^2)^{3/2}}{(y^2-1)+a^2y^4}+2a\int_{0}^{1}\frac{y^2(1-y^2)^{1/2}}{(y^2-1)^2+a^2y^4}dy=2a\left(I+J\right)\tag3$$
Integral I: Making another sub: $y=\sin(u)$, $u=\arcsin(y)$, $dy=\cos(u) du$
It is too much to write everything down, finally got to: $$I=\int_{0}^{\pi/2}\frac{cos^4(u)}{a^2\sin^4(u)+\cos^4(u)}du$$
Using trig identities we can rewrite
$$I=\int_{0}^{\pi/2}\sec^2(u)\frac{du}{(1+\tan^2(u))(1+a^2\tan^4(u))}$$
Make another sub: $s=\tan(u)$, $du=\frac{1}{\sec^2(s)}ds$
$$I=\int_{0}^{\infty}\frac{ds}{(1+s^2)(1+a^2s^4)}$$
Using partial fraction decomp:
$$I=\frac{\pi}{2(a^2+1)}-\frac{a^2}{a^2+1}\int_{0}^{\infty}\frac{s^2-1}{a^2s^4+1}ds$$
Integral J: Making another sub: $y=\sin(v)$, $v=\arcsin(y)$, $dy=\cos(v) dv$
$$J=\int_{0}^{\pi/2}\frac{\cos^2(v)\sin^2(v)}{(a^2+1)\sin^4(v)-2\sin^2(v)+1}$$
Using trig identities to rewrite
$$J=\int_{0}^{\pi/2}\sec^2(v)\cdot \frac{\tan^2(v)}{(1+\tan^2(v))(a^2\tan^4(v)+1)}$$
Make another sub: $t=\tan(v)$, $dv=\frac{1}{\sec^2(v)}dt$
$$J=\int_{0}^{\infty}\frac{t^2}{(1+t^2)(1+a^2t^4)}dt$$
Using partial fraction decomp:
$$J=\frac{1}{1+a^2}\color{red}{\int_{0}^{\infty}\frac{a^2t^2+1}{a^2t^4+1}dt}-\frac{\pi}{2(1+a^2)}$$
The red integral it is definitely way out of my reach!
$$\int_{0}^{\infty}\frac{a^2t^2+1}{a^2t^4+1}dt=\int_{0}^{\infty}\frac{t^2}{t^4+a^{-2}}dt+\int_{0}^{\infty}\frac{dt}{a^2t^4+1}$$
The above approach seem to be not helping in evaluating the question.
After simplification I got to:
$$2a(I+J)=\int_{0}^{1}\frac{\arctan\left(\frac{ax}{1-x}\right)}{\sqrt{1-x}}\frac{dx}{x^{3/2}} =\frac{1}{1+a^2}\int_{0}^{\infty}\frac{a^2+1}{a^2t^4+1}dt$$
If my work so far it is correct, then I am shruggle in solving this integral
$$\int_{0}^{\infty}\frac{1}{a^2t^4+1}dt=\int_{0}^{\infty}\frac{dt}{(at^2-i)(at^2+i)}dt$$
$\newcommand{\Im}{\operatorname{Im}}$I don't think your last integral is correct. You have made a mistake somewhere which I could not find.
Assume wolog $a\geq 0$ (because the case $a<0$ follows from the case $a>0$ by noticing that the arctangent is an odd function). Let the original integral be $F(a)$. Set $u=\frac{x}{1-x}$ to get: $$F(a)= \int^\infty_0 \frac{\arctan(au)}{u^{3/2}}\,du$$ Integrate by parts to get: $$F(a)= 2\int^\infty_0 \frac{a}{\sqrt[]{u} (1+a^2u^2)}\,du$$
Now set $t=a u$ to get:
$$F(a) = 2\sqrt[]{a} \int^\infty_0 \frac{dt}{\sqrt[]{t}(1+t^2)}$$This integral can be easily found using Residue Theorem or many other methods. For instance you can set $z=\sqrt[]{t}$ to get: $$F(a) = 4 \sqrt[]{a} \int^\infty_0 \frac{1}{1+z^4}\,dz = 2\sqrt[]{a} \int^\infty_{-\infty} \frac{1}{1+z^4}\,dz = 2\sqrt[]{a} \ \ \Im \left[ \int^\infty_{-\infty} \frac{1}{z^2-i}\,dz \right]$$ Using the Residue Theorem by enclosing the pole at $e^{i\pi/4}$ we conclude that: $$F(a) = 2\sqrt[]{a} \Im\left[ \pi i e^{-i\pi/4}\right] = \sqrt[]{2a}\ \pi$$
That was for $a>0$. In general $$F(a) = \operatorname{sign} (a) \sqrt[]{2|a|}\ \pi$$