I tried going the traditional way without fancy substitutions, unsurprisingly, I did not get anywhere beyond that annoying root.
I also went with a polar substitution where $0\leq r \leq acosec(\theta)$ and $\pi/4 \leq \theta \leq \pi/2$, but that did not work either.
How do I find my way around that annoying square root in the denominator? I can't seem to find the "hidden" simplification (if there is any)
Any help would be greatly appreciated
$$I=\int_{0}^{a}\int_{x}^{a}\frac{dy\,dx}{\sqrt{(a^2+x^2)(a-y)(y-x)}}$$Let $x=a X$ and $y=a Y$ to make $$I=\int_{0}^{1}\int_{X}^{1}\frac{dY \,dX}{\sqrt{\left(1+X^2\right) (1-Y) (Y-X)}}$$ Using Euler substitution $$J=\int\frac{dY }{\sqrt{ (1-Y) (Y-X)}}=2\tan ^{-1}\left(\sqrt{\frac{X-Y}{1-Y}}\right)$$ and everything becomes more than simple.