The integral is:$$I=\displaystyle\int_{0}^{\infty}e^{-ax^2}\cos(bx)\,dx$$
What I tried:
I was trying it with Fourier transform, so I have here: $$f(t)=e^{-at^2}\cos(bt)$$ I know that: $$\mathcal{F}\left(e^{-ax^2} \right)=\dfrac{1}{\sqrt{2a}}e^{-\frac{\omega^2}{4a}}$$ Then used frequency modulation property: $$\cos(bt)=\dfrac{1}{2}\left(e^{-jbt}+e^{jbt}\right)$$ That gives me: $$\mathcal{F}\left[f(t)\right]=\dfrac{1}{2\sqrt{2a}}\left[e^{\dfrac{-(\omega+b)^2}{4a}}+e^{\dfrac{-(\omega-b)^2}{4a}}\right]$$ For the integral I have to evaluate at $\omega=0$: $$\mathcal{F}\bigg{|}_{\omega=0}=\dfrac{1}{\sqrt{2a}}\left[e^{-\frac{b^2}{4a}}\right]$$ This is what i got but the answer is: $$I=\sqrt{\dfrac{\pi}{4a}}\left[e^{-\frac{b^2}{4a}}\right]$$ I don't know from where this $\sqrt{\dfrac{\pi}{2}}$ term is coming?
A differentiation trick works well for this problem. \begin{align} F(b) & = \int_{0}^{\infty}e^{-ax^2}\cos(bx)dx \\ F'(b) & = -\int_{0}^{\infty}e^{-ax^2}x\sin(bx)dx \\ & = \frac{1}{2a}\int_{0}^{\infty}\left(\frac{d}{dx}e^{-ax^2}\right)\sin(bx)dx \\ & = -\frac{1}{2a}\int_{0}^{\infty}e^{-ax^2}\frac{d}{dx}\sin(bx)dx \\ & = -\frac{b}{2a}\int_{0}^{\infty}e^{-ax^2}\cos(bx)dx \\ & = -\frac{b}{2a}F(b). \end{align} Therefore, there is a constant $C$ such that $$ F(b) = Ce^{-b^2/4a} $$ The constant is $F(0)=C$, which is \begin{align} F(0) & =\int_{0}^{\infty}e^{-ax^2}dx \\ & = \frac{1}{2}\int_{-\infty}^{\infty}e^{-ax^2}dx \\ & = \frac{1}{2}\left[\int_{-\infty}^{\infty}\int_{-\infty}^{\infty}e^{-a(x^2+y^2)}dxdy\right]^{1/2} \\ & = \frac{1}{2}\left[\int_{0}^{2\pi}\int_{0}^{\infty}e^{-ar^2}rdrd\theta\right]^{1/2} \\ & = \frac{\sqrt{2\pi}}{2}\left[\frac{1}{2a}\int_{0}^{\infty}e^{-ar^2}(2ar)dr\right]^{1/2} \\ & = \frac{\sqrt{2\pi}}{2}\frac{1}{\sqrt{2a}} \end{align} Therefore, $$ \int_{0}^{\infty}e^{-ax^2}\cos(bx)dx = F(b)= F(0)e^{-b^2/4a}=\frac{1}{2}\sqrt{\frac{\pi}{a}}e^{-b^2/4a}. $$