I need to calculate following definite integral $$\frac{1}{2\pi }\int_0^\infty \frac{x^2 e^{-x^2/\sigma^2 } }{\sigma} \frac{e^{-\frac{\lambda}{{ax^2+b}}}}{\sqrt{ax^2+b}} ~~dx$$
It is actually finding the expected value of $x\varphi(λ/\sqrt {ax^2+b})$ , where $\varphi$ is pdf of a standard normal distribution and $x$ is a random variable with Rayleigh distribution with parameter $\sigma$.
When $b=0$ the integral is equal to
$$\frac1{4 \pi \sigma \sqrt{a}} \int_0^{\infty} du \, e^{-\left (\frac{u}{\sigma^2} + \frac{\lambda}{a u} \right )} $$
Consider then
$$ \int_0^{\infty} du \, e^{-p \left (u + \frac{q}{u} \right )} $$
Let $v = u+q/u$; then $u^2-v u+q=0$, or
$$u = \frac{v}{2} \pm \frac12 \sqrt{v^2-4 q}$$ $$du = \frac12\left ( 1 \pm \frac{v}{\sqrt{v^2-4 q}}\right ) dv $$
The integral is then
$$\frac12\int_{\infty}^{\sqrt{q}} dv \left ( 1 - \frac{v}{\sqrt{v^2-4 q}}\right ) e^{-p v} + \frac12\int_{\sqrt{q}}^{\infty} dv \left ( 1 + \frac{v}{\sqrt{v^2-4 q}}\right ) e^{-p v}$$
or
$$\int_{\sqrt{q}}^{\infty} dv \frac{v}{\sqrt{v^2-4 q}} e^{-p v} = \int_0^{\infty} dy \, e^{-p \sqrt{y^2+4 q}} = 2 \sqrt{q} \int_0^{\infty} dt \, \cosh{t} \, e^{-2 p \sqrt{q} \cosh{t}}$$
or, the integral is
$$2 \sqrt{q} K_1 \left (2 p \sqrt{q} \right ) $$
where $K_1$ is the modified Bessel function of the second kind, of first order. Plug in $p=1/\sigma^2$ and $q=\sigma^2 \lambda/a$ and you are done.