Can I transform this integral $\int\int dx dy f(x) g(x,y) f(y)$ into the Fourier space? I am expecting something like $\int\int dk dk'f(k)g(k,k')f(k') $, although I can't prove they are equivalent.
Note: $f(k)$ is the Fourier transform of $ f(x)$. $g(k,k') $ is the Fourier transform of $ g(x,y)$
Assuming we use the unitary Fourier transform, \begin{multline} \int_{-\infty}^\infty f(x)g(x)dx = \frac{1}{2\pi}\int\!\!\!\int\!\!\!\int_{-\infty}^\infty F(k)G(k')e^{i(k+k')x}dk\,dk'\,dx \\ =\frac{1}{2\pi}\int\!\!\!\int_{-\infty}^\infty F(k)G(k')\left[\int_{-\infty}^\infty e^{i(k+k')x}dx\right]\,dk\,dk' \\= \int\!\!\!\int_{-\infty}^\infty F(k)G(k')\delta(k+k')dk\,dk' = \int_{-\infty}^\infty F(k)G(-k)dk \end{multline} So, generalizing this to your double integral gives $$ \int\!\!\!\int_{-\infty}^\infty f(x)g(x,y)f(y)dx\,dy = \int\!\!\!\int_{-\infty}^\infty F(k)G(-k,-k')F(k')dk\,dk' $$