Let $f(x) = \exp(-x^2/2)$ and let $F(x) = f^\wedge = \int_{-\infty}^\infty f(t) \exp(-ixt) dt$ be the Fourier transform of $f$. It can be shown that $F$ satisfied the ODE
$$y'(x) = - x y(x)$$
(see also this post).
Furthermore, the general solution of the ODE is
$$y(x) = C \exp(-x^2/2)$$
for any constant $C$.
In my exercise, I am asked to show that $F(x) = \sqrt{2 \pi} \exp(-x^2/2)$. We were told to use the identity
$$\int_{-\infty}^\infty \exp(-x^2/2) dx = \sqrt{2\pi}$$.
I tried to write
$$C \exp(-x^2/2) = f^\wedge(x) = \int_{-\infty}^\infty \exp(-t^2/2 - ixt) dt,$$
but the right term does not seem to converge. Can anyone help me here?
Once you have the general solution, $y(x)=Ce^{-x^2/2}$, to the ODE $y'(x)=-xy(x)$, the constant $C$ can be found as follows.
Recall that $y(x)$ is defined here as
$$y(x)=\int_{-\infty}^\infty e^{-t^2/2}e^{-ixt}\,dt$$
So, we see that $C=y(0)=\int_{-\infty}^\infty e^{-t^2/2}\,dt=\sqrt{2\pi}$. Therefore, $C=\sqrt{2\pi}$ and we are done!