Let $f(t)=\int_{0}^{1}\sqrt\omega e^{\omega^2}cos(\omega t)d\omega$.
I want to compute $\int_{-\infty}^{\infty}|f'(t)|^2 dt = I$.
By Plancherel formula, $\int_{-\infty}^{\infty}|f'(t)|^2dt = \frac{1}{2\pi}\int_{-\infty}^{\infty}\hat{f'}.\overline{\hat{f'}} \ dt$.
On the other hand, $\hat{f'}.\overline{\hat{f'}} = it\hat{f}(-it\hat{f})=t^2 (\hat{f})^2$.
Hence, it suffices to compute $\hat{f}$. In order to do so, I tried to use the Fourier Transform Inverse Formula, however I got stucked.
Any suggestions or help, please ? (in order to compute $I$)
You simply have to swap the integrals:
\begin{align} \hat f(x) &= \frac{1}{\sqrt{2\pi}}\int_{-\infty}^{+\infty} e^{-itx}\int_0^1 \sqrt{w}e^{w^2} \cos(wt) dw dt \\ &= \frac{1}{\sqrt{2\pi}} \int_0^1 \sqrt{w}e^{w^2}\int_{-\infty}^{+\infty} \cos(wt)e^{-itx} dt dw \\ &= \frac{1}{\sqrt{2\pi}} \int_0^1 \sqrt{w}e^{w^2} \frac{\sqrt{2\pi}}{2}\Big(\delta(x-w) +\delta(x+w)\Big) dw \\ &= \frac{1}{2} (\sqrt{x} + \sqrt{-x})e^{x^2} \mathbf{1}_{[0,1]}(x)\\ &= \frac{1+i}{2}\sqrt{x}e^{x^2} \mathbf{1}_{[0,1]}(x) \end{align}
Where $ \mathbf{1}_{[0,1]}(x)$ is the indicator function on $[0,1]$. Equivalently you could write $H(x(1-x)) = \mathbf{1}_{[0,1]}(x)$ using the Heaviside step function.
Finally you can find your integral using $\int x^3 e^{2x^2}dx = \frac{1}{8}e^{2x^2}(2x^2 -1) +C$