Given that $\int_{-\infty}^{\infty} e^{(-\frac{1}{2}x^2)}dx = \sqrt{2\pi}$,
Find $I(a,b) = \int_{-\infty}^{\infty} e^{-\frac{1}{2}(ax^2+\frac{b}{x^2})}dx$.
The question suggests using the substitution: $$y = \frac{1}{2}(\sqrt{a}x-\frac{\sqrt{b}}{x})$$ if careful attention is paid to the limits. I'm not sure how the limits of integral changes as I can't see why the limits wouldn't remain the same using the substitution.
On
The key here is to note that this integral is improper at zero, so before performing the substitution notice the integrand is even so we may rewrite the integral as $$I(a,b)=2\int_{0+}^\infty e^{-\frac{1}{2}(ax^2+\frac{b}{x^2})}dx$$ Let $y=\sqrt{a}x-\frac{\sqrt{b}}{x}$. Note the bounds become $-\infty$ to $\infty$, since as $x\rightarrow 0^+$, $y\rightarrow -\infty$. Also notice $y^2=ax^2+\frac{b}{x^2}-2\sqrt{ab}$, and $$dy=\sqrt{a}+\frac{\sqrt{b}}{x^2}=\frac{\sqrt{y^2+4\sqrt{ab}}-y}{2\sqrt{b}}\sqrt{y^2+4\sqrt{ab}}\ dx$$ so the integral is $$I(a,b)=2\int_{-\infty}^\infty e^{-\frac{1}{2}(y^2+2\sqrt{ab})}\frac{2\sqrt{b}}{\sqrt{y^2+4\sqrt{ab}}}\frac{1}{\sqrt{y^2+4\sqrt{ab}}-y}dy$$ Combining the positive and negative sides of the axis gives $$I(a,b)=2\int_{0}^\infty e^{-\frac{1}{2}(y^2+2\sqrt{ab})}\frac{2\sqrt{b}}{\sqrt{y^2+4\sqrt{ab}}}\left(\frac{1}{\sqrt{y^2+4\sqrt{ab}}-y}+\frac{1}{\sqrt{y^2+4\sqrt{ab}}+y}\right)dy$$ $$=\frac{2}{\sqrt{a}}\int_0^\infty e^{-\frac{1}{2}(y^2+2\sqrt{ab})}dy$$ And from there you can use your identity.
The integrand is even, so restrict the integration range to $x\ge 0$ and introduce a factor of $2$. The only discontinuity is at $x=0$, and to the right of that $dy/dx>0$. If you're still stuck, see here.