I am willing to compute the Fourier transformation of the following function:
$$ \Phi(r) = (I\Delta - \nabla \nabla )[r\operatorname{erf}(\xi r)] $$
Where, $r = X-X_0$, $\xi$ is a positive constant, $\phi$ is a $3\times 3$ matrix and $I\Delta$ is a diagonal matrix with laplacian operator on the diagonal and $\nabla \nabla$ is a matrix which has $\frac{\partial^2}{\partial x_i \partial x_j}$ in its $i$th row and $j$th column.
To compute the Fourier transformation one needs to compute:
$$ \hat{\Phi}(k,\hat{x_0}) = \int_{\mathbb{R}^3} \exp(ik\cdot X)(I\Delta - \nabla \nabla )[r\ \operatorname{erf}(\xi r)]d^3X $$
where $k$ is a three dimensional phase vector and $i= \sqrt{-1}$. The following step in computing this integral is unclear for me:
$$ \int_{\mathbb{R}^3} \exp(ik\cdot X)(I\Delta - \nabla \nabla )[r\operatorname{erf}(\xi r)]d^3X = (-I|k|^2+kk^T)\int_{\mathbb{R}^3} \exp(ik\cdot X)r\operatorname{erf}(\xi r)d^3X $$
My guess is that they somehow moved the differential operators from $r\operatorname{erf}(\xi r)$ to $\exp(ik\cdot X)$. But can someone please help me understand how is this possible?
Moving the differential operator from one factor to the other under the integral sign is just integration by parts. In multiple dimensions, this is also called the Green's identity.
http://en.wikipedia.org/wiki/Green%27s_identities
Let's write
$$f=\exp(ik\cdot X)$$ and $$\vec{v}=r\operatorname{erf}(\xi r)$$
You are integrating
$$f(\partial_j\partial_j v_i-\partial_i \partial_j v_j)$$ where I used the summation convention (imagine sums over the repeated index $j$). Index notation is almost necessary, because we are working with combinations of indices and derivatives that are hard to write with $\nabla$.
To extract the second term, write $$\partial_j(f \partial_i v_j)=(\partial_j f)(\partial_i v_j)+f(\partial_i\partial_j v_j)$$ For the first term: $$\partial_j(f\partial_j v_i)=(\partial_j f)(\partial_j v_i)+f(\partial_j\partial_j v_i)$$
You can now write your integrand as: $$f(\partial_j\partial_j v_i-\partial_i \partial_j v_j)= \partial_j(f\partial_j v_i-f \partial_i v_j)-(\partial_j f)(\partial_j v_i)+(\partial_j f)(\partial_i v_j) $$
The first term is something we wish to cancel out or send to zero. Last two still need a bit of work:
$$\partial_j(f(\partial_j v_i))=(\partial_j\partial_j f)v_i+(\partial_j f)(\partial_j v_i)$$ $$\partial_j(f(\partial_i v_j))=(\partial_j\partial_i f)v_j+(\partial_i f)(\partial_j v_j)$$
Rearrange and plug into the expression above: $$f(\partial_j\partial_j v_i-\partial_i \partial_j v_j)= \partial_j(f\partial_j v_i-f \partial_i v_j)-\partial_j(f(\partial_j v_i))+(\partial_j\partial_j f)v_i +\partial_j(f(\partial_i v_j))-(\partial_j\partial_i f)v_j =(\partial_j\partial_j f)v_i-(\partial_j\partial_i f)v_j$$
This is exactly what you wanted! Using $f=(\exp{ik\cdot X})$, you get $$=(-k_j k_j f)v_i-(-k_j k_i f)v_j=-(|k|^2I-\vec{k}\otimes \vec{k})f\vec{v}$$
p.s. did you get this operator from a double curl?