Assuming that $a$ and $b$ are positive real numbers. I'm trying to find a closed-form expression for the following definite integral. \begin{equation} I(a,b) = \int_{-\infty}^{\infty} \left((x^2+a)(x^2+b)\right)^{\tfrac{-3}{4}} dx. \end{equation}
Do you know how can I derive the closed-form value of $I$?
Define the function $\mathcal{I}:\mathbb{R}_{>0}^{2}\rightarrow\mathbb{R}$ via the improper integral
$$\mathcal{I}{\left(a,b\right)}:=\int_{-\infty}^{\infty}\mathrm{d}x\,\left[\left(x^{2}+a\right)\left(x^{2}+b\right)\right]^{-3/4}.$$
Obviously, $\mathcal{I}$ is symmetric in its two variables:
$$\mathcal{I}{\left(a,b\right)}=\mathcal{I}{\left(b,a\right)};~~~\small{(a,b)\in\mathbb{R}_{>0}^{2}}.$$
We can assume then without loss of generality that $0<a\le b$.
It can also be shown that $\mathcal{I}$ obeys the scaling relation
$$\mathcal{I}{\left(a,b\right)}=s\mathcal{I}{\left(sa,sb\right)};~~~\small{(a,b,s)\in\mathbb{R}_{>0}^{3}}.$$
Using this scaling relation we can set at least one parameter to unity:
$$\mathcal{I}{\left(a,b\right)}=b^{-1}\mathcal{I}{\left(\frac{a}{b},1\right)}.$$
So it suffices to evaluate $\mathcal{I}{\left(c,1\right)}$ for $0<c\le1$.
In the case of equal parameters, we have
$$\begin{align} \mathcal{I}{\left(1,1\right)} &=\int_{-\infty}^{\infty}\mathrm{d}x\,\left[\left(x^{2}+1\right)^{2}\right]^{-3/4}\\ &=\int_{-\infty}^{\infty}\mathrm{d}x\,\left(x^{2}+1\right)^{-3/2}\\ &=\int_{-\infty}^{\infty}\mathrm{d}x\,\frac{d}{dx}\left[\frac{x}{\sqrt{x^{2}+1}}\right]\\ &=\lim_{x\to\infty}\frac{x}{\sqrt{x^{2}+1}}-\lim_{x\to-\infty}\frac{x}{\sqrt{x^{2}+1}}\\ &=2.\\ \end{align}$$
Turning to the case of unequal parameters, suppose $c\in(0,1)$. Consider the function $g:\mathbb{R}_{>0}\rightarrow\mathbb{R}$ given by
$$g{\left(x\right)}:=\frac{\sqrt[4]{\left(x^{2}+c\right)\left(x^{2}+1\right)}}{x}.$$
Note that $\forall x>0:g{(x)}>1$. Also take note of the following limits, which are straightforward to compute:
$$\lim_{x\to0^{+}}g{\left(x\right)}=\infty,$$
$$\lim_{x\to\infty}g{\left(x\right)}=1.$$
Finding the derivative of $g$, it's easy to see that it is strictly negative:
$$g^{\prime}{\left(x\right)}=-\frac{\left(c+1\right)x^{2}+2c}{2x^{2}\left[\left(x^{2}+c\right)\left(x^{2}+1\right)\right]^{3/4}}<0;~~~\small{x>0}.$$
Thus, $g$ is a strictly decreasing bijection from $\mathbb{R}_{>0}$ to $\left(1,\infty\right)$ with a differentiable inverse.
We will see that the integral $\mathcal{I}{\left(c,1\right)}$ can be transformed into an elliptic integral using the substitution relation
$$g{\left(x\right)}=\frac{\sqrt[4]{\left(x^{2}+c\right)\left(x^{2}+1\right)}}{x}=y\iff x=g^{-1}{\left(y\right)}.$$
It follows that
$$\sqrt[4]{\left(x^{2}+c\right)\left(x^{2}+1\right)}=xy,$$
$$\implies\left(x^{2}+c\right)\left(x^{2}+1\right)=x^{4}y^{4},$$
$$\implies x^{4}+\left(c+1\right)x^{2}+c=x^{4}y^{4},$$
$$\implies0=\left(y^{4}-1\right)x^{4}-\left(c+1\right)x^{2}-c,$$
$$\implies x^{2}=\frac{c+1\pm\sqrt{(c+1)^{2}+4c(y^{4}-1)}}{2(y^{4}-1)},$$
$$\implies x^{2}=\frac{c+1\pm\sqrt{4cy^{4}+\left(c-1\right)^{2}}}{2(y^{4}-1)},$$
$$\implies x^{2}=\frac{\left[c+1\pm\sqrt{4cy^{4}+\left(c-1\right)^{2}}\right]\left[c+1\mp\sqrt{4cy^{4}+\left(c-1\right)^{2}}\right]}{2(y^{4}-1)\left[c+1\mp\sqrt{4cy^{4}+\left(c-1\right)^{2}}\right]},$$
$$\implies x^{2}=\frac{(-1)4c\left(y^{4}-1\right)}{2(y^{4}-1)\left[c+1\mp\sqrt{4cy^{4}+\left(c-1\right)^{2}}\right]},$$
$$\implies x^{2}=\frac{2c}{-c-1\pm\sqrt{4cy^{4}+\left(c-1\right)^{2}}},$$
Since $\frac{2c}{-c-1-\sqrt{4cy^{4}+\left(c-1\right)^{2}}}<0<\frac{2c}{-c-1+\sqrt{4cy^{4}+\left(c-1\right)^{2}}}$ for $0<c\land1<y$, the correct root for $x^2$ is then
$$x^{2}=\frac{2c}{\sqrt{4cy^{4}+\left(c-1\right)^{2}}-c-1}.$$
Differentiating both sides w.r.t. $y$, we have
$$2x\frac{dx}{dy}=-\frac{16c^{2}y^{3}}{\left[\sqrt{4cy^{4}+\left(c-1\right)^{2}}-c-1\right]^{2}\sqrt{4cy^{4}+\left(c-1\right)^{2}}}.$$
When we apply this substitution to $\mathcal{I}$, we find the integral can be rewritten as
$$\begin{align} \mathcal{I}{\left(c,1\right)} &=\int_{-\infty}^{\infty}\mathrm{d}x\,\left[\left(x^{2}+c\right)\left(x^{2}+1\right)\right]^{-3/4}\\ &=2\int_{0}^{\infty}\mathrm{d}x\,\left[\left(x^{2}+c\right)\left(x^{2}+1\right)\right]^{-3/4};~~~\small{\text{even symmetry}}\\ &=\int_{0}^{\infty}\mathrm{d}x\,\frac{2}{\left[\sqrt[4]{\left(x^{2}+c\right)\left(x^{2}+1\right)}\right]^{3}}\\ &=\int_{0}^{\infty}\mathrm{d}x\,\frac{2}{\left[xg{\left(x\right)}\right]^{3}}\\ &=\int_{0}^{\infty}\mathrm{d}x\,\frac{2x}{x^{4}\left[g{\left(x\right)}\right]^{3}}\\ &=\int_{+\infty}^{1}\mathrm{d}y\,\frac{\left[\sqrt{4cy^{4}+\left(c-1\right)^{2}}-c-1\right]^{2}}{4c^{2}}\cdot\frac{2x}{y^{3}}\,\frac{dx}{dy};~~~\small{\left[g{\left(x\right)}=y\right]}\\ &=\int_{1}^{\infty}\mathrm{d}y\,\frac{4}{\sqrt{4cy^{4}+\left(c-1\right)^{2}}},\\ \end{align}$$
which is of course an elliptic integral.
Setting $p:=\frac{\sqrt{1-c}}{\sqrt{2}\sqrt[4]{c}}$, note that $0<p$ for $0<c<1$. Then,
$$\begin{align} \mathcal{I}{\left(c,1\right)} &=\int_{1}^{\infty}\mathrm{d}y\,\frac{4}{\sqrt{4cy^{4}+\left(c-1\right)^{2}}}\\ &=\int_{0}^{1}\mathrm{d}t\,\frac{4}{\sqrt{4c+\left(c-1\right)^{2}t^{4}}};~~~\small{\left[y=t^{-1}\right]}\\ &=\frac{2}{\sqrt{c}}\int_{0}^{1}\mathrm{d}t\,\frac{1}{\sqrt{1+\left(\frac{\sqrt{1-c}}{\sqrt{2}\sqrt[4]{c}}\right)^{4}t^{4}}}\\ &=\frac{2}{\sqrt{c}}\int_{0}^{1}\mathrm{d}t\,\frac{1}{\sqrt{1+p^{4}t^{4}}}\\ &=\frac{2}{\sqrt{c}}\int_{0}^{p}\mathrm{d}u\,\frac{p^{-1}}{\sqrt{1+u^{4}}};~~~\small{\left[pt=u\right]}\\ &=\frac{2}{p\sqrt{c}}\int_{0}^{p}\mathrm{d}u\,\frac{1}{\sqrt{1+u^{4}}}\\ &=\frac{\sqrt{2}}{\sqrt{1-c}\sqrt[4]{c}}\int_{0}^{p}\mathrm{d}u\,\frac{2}{\sqrt{1+u^{4}}}\\ &=\frac{\sqrt{2}}{\sqrt{1-c}\sqrt[4]{c}}\,\mathcal{J}{\left(p\right)},\\ \end{align}$$
where in the last line above we've introduced the auxiliary function $\mathcal{J}:\mathbb{R}_{>0}\rightarrow\mathbb{R}$ given by the elliptic integral
$$\mathcal{J}{\left(p\right)}:=\int_{0}^{p}\mathrm{d}u\,\frac{2}{\sqrt{1+u^{4}}}.$$
For $p>0$, we have $-1<\frac{1-p^{2}}{1+p^{2}}<1\implies0<\frac{\pi}{2}-\arcsin{\left(\frac{1-p^{2}}{1+p^{2}}\right)}=2\arctan{\left(p\right)}<\pi$. Then,
$$\begin{align} \mathcal{J}{\left(p\right)} &=\int_{0}^{p}\mathrm{d}u\,\frac{2}{\sqrt{1+u^{4}}}\\ &=\int_{0}^{p^{2}}\mathrm{d}v\,\frac{2}{2\sqrt{v}\sqrt{1+v^{2}}};~~~\small{\left[u=\sqrt{v}\right]}\\ &=\int_{0}^{p^{2}}\mathrm{d}v\,\frac{1}{\sqrt{v\left(1+v^{2}\right)}}\\ &=\int_{1}^{\frac{1-p^{2}}{1+p^{2}}}\mathrm{d}w\,\frac{(-2)}{\left(1+w\right)^{2}}\cdot\frac{1}{\sqrt{\left(\frac{1-w}{1+w}\right)\left[1+\left(\frac{1-w}{1+w}\right)^{2}\right]}};~~~\small{\left[v=\frac{1-w}{1+w}\right]}\\ &=\int_{\frac{1-p^{2}}{1+p^{2}}}^{1}\mathrm{d}w\,\frac{2}{\sqrt{\left(1+w\right)^{4}\left(\frac{1-w}{1+w}\right)\left[1+\left(\frac{1-w}{1+w}\right)^{2}\right]}}\\ &=\int_{\frac{1-p^{2}}{1+p^{2}}}^{1}\mathrm{d}w\,\frac{2}{\sqrt{\left(1+w\right)\left(1-w\right)\left[\left(1+w\right)^{2}+\left(1-w\right)^{2}\right]}}\\ &=\int_{\frac{1-p^{2}}{1+p^{2}}}^{1}\mathrm{d}w\,\frac{2}{\sqrt{\left(1-w^{2}\right)\left(2+2w^{2}\right)}}\\ &=\sqrt{2}\int_{\frac{1-p^{2}}{1+p^{2}}}^{1}\mathrm{d}w\,\frac{1}{\sqrt{1-w^{4}}}\\ &=\sqrt{2}\int_{\arcsin{\left(\frac{1-p^{2}}{1+p^{2}}\right)}}^{\arcsin{\left(1\right)}}\mathrm{d}\omega\,\frac{\cos{\left(\omega\right)}}{\sqrt{1-\sin^{4}{\left(\omega\right)}}};~~~\small{\left[w=\sin{\left(\omega\right)}\right]}\\ &=\sqrt{2}\int_{\arcsin{\left(\frac{1-p^{2}}{1+p^{2}}\right)}}^{\frac{\pi}{2}}\mathrm{d}\omega\,\frac{\cos{\left(\omega\right)}}{\sqrt{\left[1-\sin^{2}{\left(\omega\right)}\right]\left[1+\sin^{2}{\left(\omega\right)}\right]}}\\ &=\sqrt{2}\int_{\arcsin{\left(\frac{1-p^{2}}{1+p^{2}}\right)}}^{\frac{\pi}{2}}\mathrm{d}\omega\,\frac{\cos{\left(\omega\right)}}{\sqrt{\cos^{2}{\left(\omega\right)}\left[2-\cos^{2}{\left(\omega\right)}\right]}}\\ &=\sqrt{2}\int_{\arcsin{\left(\frac{1-p^{2}}{1+p^{2}}\right)}}^{\frac{\pi}{2}}\mathrm{d}\omega\,\frac{\cos{\left(\omega\right)}}{\left|\cos{\left(\omega\right)}\right|\sqrt{2\left[1-\frac12\cos^{2}{\left(\omega\right)}\right]}}\\ &=\int_{\arcsin{\left(\frac{1-p^{2}}{1+p^{2}}\right)}}^{\frac{\pi}{2}}\mathrm{d}\omega\,\frac{1}{\sqrt{1-\frac12\cos^{2}{\left(\omega\right)}}}\\ &=\int_{0}^{\frac{\pi}{2}-\arcsin{\left(\frac{1-p^{2}}{1+p^{2}}\right)}}\mathrm{d}\varphi\,\frac{1}{\sqrt{1-\frac12\cos^{2}{\left(\frac{\pi}{2}-\varphi\right)}}};~~~\small{\left[\omega=\frac{\pi}{2}-\varphi\right]}\\ &=\int_{0}^{2\arctan{\left(p\right)}}\mathrm{d}\varphi\,\frac{1}{\sqrt{1-\frac12\sin^{2}{\left(\varphi\right)}}}\\ &=F{\left(2\arctan{\left(p\right)},\frac{1}{\sqrt{2}}\right)},\\ \end{align}$$
where the elliptic integral of the first kind $F$ is defined by
$$F{\left(\theta,k\right)}:=\int_{0}^{\theta}\mathrm{d}\varphi\,\frac{1}{\sqrt{1-k^{2}\sin^{2}{\left(\varphi\right)}}};~~~\small{k\in(-1,1)\land\theta\in\mathbb{R}}.$$
Putting everything together, we obtain the following result: