Those integral techniques I thought I'd never need to remember are coming back to bite me.
Is there a way to compute integrals of the form $$ \int_0^\infty \frac1{\sqrt{1 + a x^2 + b x^4}} dx $$
I am feeling hopeless about this, but any reference/hints will be appreciated.
Perhaps you still don't need to remember them, because they don't help much with this integral. OK, substitution helps a bit. But, the integral cannot be expressed in terms of functions that you are familiar with. It can be only expressed in terms of the complete elliptic integral of first kind, denoted by $K$ below.
If you remember the quadratic formula, you will not be surprised to learn that the sign of $a^2-4b$ matters. (By the way, I assume that $a,b$ in your post are positive numbers.)
Case 1: $a^2\ge 4b$. The substitution $u=b^{1/4}x$ changes the integral to $$b^{-1/4}\int_0^\infty \frac1{\sqrt{1+cu^2+u^4}}\,du \label{1}\tag{1}$$ where $c=a/\sqrt{b}\ge 2$. We can write $c=k+k^{-1}$ for some $0<k\le 1$ and get $$\int_0^\infty \frac1{\sqrt{1+cu^2+u^4}}\,du = \sqrt{k}\,K\left(\sqrt{1-k^2}\right) \label{2}\tag{2}$$ courtesy of Maple. (I don't see right away which substitution changes the left side of $(\ref{2})$ to the definition of $K$.)
Case 2: $a^2\le 4b$. Substitute $u=a^{1/2}x$ to change the integral to $$a^{-1/2}\int_0^\infty \frac1{\sqrt{1+u^2+cu^4}}\,du \label{3}\tag{3}$$ where $c=b/a^2\ge 1/4$. Here, $$\int_0^\infty \frac1{\sqrt{1+u^2+cu^4}}\,du = c^{-1/4}K\left(\frac12 \sqrt{2-1/\sqrt{c}}\right) \label{4}\tag{4}$$ Again, I fail to justify $(\ref{4})$ with an explicit transformation.
Sanity check: when $a^2=4b$, the first formula yields $b^{-1/4}K(0)$, while the second gives $a^{-1/2}(b/a^2)^{-1/4}K(0)$. Same thing.