In my book of physics, there is a derivation using some properties of Fourier transform I don't understand.
It is written the following :
$$\frac{1}{2 \epsilon_0} \int d^3 k \frac{\rho^*(\vec{k},t)\rho(\vec{k},t)}{k^2}=\frac{1}{8 \pi \epsilon_0} \int d^3 r \int d^3r' \frac{\rho(\vec{r},t)\rho(\vec{r'},t)}{|\vec{r}-\vec{r'}|}$$
The convention of the Fourier transform are the following :
$$f(\vec{r})=\frac{1}{(2 \pi)^{\frac{3}{2}}} \int f(\vec{k}) e^{+i \vec{k}\cdot\vec{r}}d^3k$$
$$f(\vec{k})=\frac{1}{(2 \pi)^{\frac{3}{2}}} \int f(\vec{r}) e^{-i \vec{k}\cdot\vec{r}}d^3r$$
How to prove the equality ?
Let $\hat \rho$ be the Fourier Transform of $\rho$ as given by
$$\hat\rho(\vec k,t)=\frac1{(2\pi)^{3/2}}\int_{\mathbb{R}^3}\rho(\vec r,t)e^{-i\vec k\cdot\vec r}\,d^3r \tag 1$$
with the inverse transform written
$$\rho(\vec r,t)=\frac1{(2\pi)^{3/2}}\int_{\mathbb{R}^3}\hat\rho(\vec k,t)e^{i\vec k\cdot\vec r}\,d^3k\tag 2$$
Let the function $f(\vec r)$ be given by
$$f(\vec r)=\int_{\mathbb{R}^3}\frac{\hat\rho(\vec k)\hat\rho^*(\vec k)}{k^2}\,e^{i\vec k\cdot\vec r}\,d^3k \tag 3$$
Taking the Laplacian of $(3)$ we find that
$$-\nabla^2 f(\vec r)=\int_{\mathbb{R^3}} \hat\rho(\vec k) \hat\rho^*(\vec k) e^{i\vec k\cdot \vec r}\, dk^3$$
Using the complex conjugate of $(2)$ for $\hat\rho^*$ in $(3)$, changing the order of integration, and using $(1)$ with $\vec r$ replaced with $\vec r+\vec r'$ reveals
$$-\nabla^2 f(\vec r)=\int_{\mathbb{R}^3}\rho(\vec r')\rho(\vec r+\vec r')\,d^3r'\tag 4$$
Integrating $(4)$, setting $\vec r=0$ and translating dummy coordinates yields
$$f(0)=\frac1{4\pi} \int_{\mathbb{R}^3}\rho(\vec r') \int_{\mathbb{R}^3}\frac{\rho(\vec r)}{|\vec r-\vec r'|}\,d^3r\, d^3 r'$$
as was to be shown!