We need to solve above integral which I obtained after manipulation. Now I used $r^2 = u $ to get:
$$\int_0^\infty \frac{bdu}{2 (u+b^2)^{(3/2)}(u+a^2)^{(1/2)}}$$ which is converted to:
$$\int_0^\infty \frac{bdu}{2 (u+b^2) \sqrt{\left(u+\frac{a^2+b^2}{2}\right)^2 - \left(\frac{a^2-b^2}{2}\right)^2}}$$
Now I am having trouble reducing it. Please give only a hint! Thank you a lot!
HINT:
Denote $I$ the given integral. Integration by parts leads to a function and an integral that is a constant multiple of $I.$
Solution (not accomplished, based on the above hint)
For $$I(b,a)=\int_{r=0}^{\infty} \frac{b r\; dr}{(r^2+b^2)^{(3/2)}(r^2+a^2)^{(1/2)}}$$ apply integration by parts considering $u(r)=\frac{1}{(r^2+a^2)^{1/2}}$ and $v '(r)=\frac{br}{(r^2+b^2)^{3/2}}.$
We get $$I=\left[\frac{-b}{\sqrt{(r^2+a^2)(r^2+b^2)}}\right]_{0}^{\infty}-\frac{b}{a}\int_{r=0}^{\infty} \frac{a r\; dr}{(r^2+b^2)^{(1/2)}(r^2+a^2)^{(3/2)}} $$ $$I(b,a)=\left[0-\frac{-b}{|ab|}\right]-\frac{b}{a}I(a,b).$$ From this we can compute $I(b,a)+\frac{b}{a}I(a,b),$ but just $I(b,a)$ is wanted.
SOLUTION with Euler's substitution
$$I=\int_{0}^{\infty} \frac{b r\; dr}{(r^2+b^2)^{(3/2)}(r^2+a^2)^{(1/2)}}=\int_{0}^{\infty} \frac{b r\; dr}{(r^2+b^2)^{2}\cdot {\sqrt{\frac{r^2+a^2}{r^2+b^2}}}}$$
Set $t=\sqrt{\frac{r^2+a^2}{r^2+b^2}},$ then $r^2=\frac{a^2-b^2t^2}{t^2-1},$ consequently $rdr=\frac{t(b^2-a^2)}{(t^2-1)^2}dt$ and $r^2+b^2=\frac{a^2-b^2}{t^2-1}.$ We obtain $$I=\int_{|\frac{a}{b}|}^1 \frac{b}{b^2-a^2}dt,$$ easy to finish.