I'm interested in understanding the sequence of substitutions needed to obtain \begin{align} \int\frac{dx}{\left(a^2+x^2\right) \sqrt{a^2+b^2+x^2}} =\frac{\tan ^{-1}\left(\frac{b x}{a \sqrt{a^2+b^2+x^2}}\right)}{a b}\, , \end{align} with $a,b>0$.
The obvious substitution \begin{align} x=\sqrt{a^2+b^2}\tan\xi \end{align} gets rid of the square root but complicates the $a^2+x^2$ factor: \begin{align} \int\frac{dx}{\left(a^2+x^2\right) \sqrt{a^2+b^2+x^2}} = \int \frac{d\xi}{ (a^2+ (a^2+b^2)\tan^2\xi)\cos\xi} \tag{1} \end{align} which doesn't seem to be immediately helpful.
The substitution $a^2+x^2=b^2\tan^2\xi$ makes things worse since $dx$ will convert to $b^2 \tan\xi d\xi/(\cos^2\xi\sqrt{b^2\tan^2\xi-a^2})$ with no realistic chance of eliminating the square root.
That would leave me with Euler's substitution \begin{align} \sqrt{a^2+b^2+x^2}=x+t \end{align} but before I get there I wonder if there's a more "obvious" way.
Your first choice is a better start than it seems, because Bioche's rules advise continuing with$$u=\sin\xi=\frac{\sqrt{a^2+b^2}\tan\xi}{\sqrt{a^2+b^2}\sec\xi}=\frac{x}{\sqrt{a^2+b^2+x^2}}.$$Then$$\frac{d\xi}{(a^2+ (a^2+b^2)\tan^2\xi)\cos\xi}=\frac{\cos\xi d\xi}{a^2\cos^2\xi+(a^2+b^2)\sin^2\xi}=\frac{du}{a^2+b^2u^2}.$$