I'm trying to evaluate an integral of the form:
$$I\equiv \int_{-\infty}^{\infty}dx\space f(x)\int_{-\infty}^{\infty}dy\space h(y)\int_{-\infty}^{\infty}d\omega\space\omega e^{i\omega(x-y)}$$
with $f(x)$ and $h(y)$ analytic functions of $x$ and $y$, respectively.
I have two separate questions that will be presented as I go over what I did.
Option 1:
Noting that
$$\int_{-\infty}^{\infty}d\omega\space\omega e^{i\omega(x-y)}=2\pi i\frac{d}{dx}\delta(x-y)$$
we can go on and write $$I=2\pi i\int_{-\infty}^{\infty}dy\space h(y)\int_{-\infty}^{\infty}dx\space f(x)\frac{d}{dx}\delta(x-y)$$ $$=-2\pi i\int_{-\infty}^{\infty}dy\space h(y)\int_{-\infty}^{\infty}dx\space \frac{d}{dx}f(x)\delta(x-y)\color{lightgrey}{+2\pi i\space h(x)f(x)\vert^{\infty}_{-\infty}}$$ $$=-2\pi i\int_{-\infty}^{\infty}dx\space h(x)\frac{d}{dx}f(x)\color{lightgrey}{+2\pi i \space h(x)f(x)\vert^{\infty}_{-\infty}}$$
where I used integration by parts,
$$\int_{-\infty}^{\infty}dx\space f(x)\frac{d}{dx}\delta(x-y)=f(x)\delta(x-y)\vert^{\infty}_{-\infty}-\int_{-\infty}^{\infty}dx\space \frac{d}{dx}f(x)\delta(x-y)$$
and the term in grey corresponds to my first question.
First question: Is it correct to set $$\int_{-\infty}^{\infty} dy\space h(y)(f(x)\delta(x-y)\vert^{\infty}_{-\infty})=0$$ , arguing that $\int_{-\infty}^{\infty}=\lim_{a\to\infty}\int_{-a}^{a}$ and using the fact the integrand vanishes for any finite $y$?
Or rather, the fact that this is integrated over $y$ from $-\infty$ to $\infty$ yields something like $$\int_{-\infty}^{\infty} dy\space h(y)(f(x)\delta(x-y)\vert^{\infty}_{-\infty})=h(x)f(x)\vert^{\infty}_{-\infty}$$ ? For this case, I added the grey terms. These terms disappear if the boundary term vanishes, that is the first case I presented. As I'm not convinced which one is correct, I will carry the grey terms along, remembering their source. However, this won't affect the next question I'm about to get to.
Option 2:
Now, let's repeat this calculation by replacing the roles of $x$ and $y$. That is, noting that:
$$\int_{-\infty}^{\infty}d\omega\space\omega e^{i\omega(x-y)}=-2\pi i\frac{d}{dy}\delta(x-y)$$
we can go on and write $$I=-2\pi i\int_{-\infty}^{\infty}dx\space f(x)\int_{-\infty}^{\infty}dy\space h(y)\frac{d}{dy}\delta(x-y)$$ $$=2\pi i\int_{-\infty}^{\infty}dx\space f(x)\int_{-\infty}^{\infty}dy\space \frac{d}{dy}h(y)\delta(x-y)\color{lightgrey}{-2\pi i\space h(x)f(x)\vert^{\infty}_{-\infty}}$$ $$=2\pi i\int_{-\infty}^{\infty}dx\space f(x)\frac{d}{dx}h(x)\color{lightgrey}{-2\pi i\space h(x)f(x)\vert^{\infty}_{-\infty}}$$
To sum up, I'll denote the two results by: $$S_1=-2\pi i\int_{-\infty}^{\infty}dx\space h(x)\frac{d}{dx}f(x)\color{lightgrey}{+2\pi i \space h(x)f(x)\vert^{\infty}_{-\infty}}$$ $$S_2=2\pi i\int_{-\infty}^{\infty}dx\space f(x)\frac{d}{dx}h(x)\color{lightgrey}{-2\pi i\space h(x)f(x)\vert^{\infty}_{-\infty}}$$
Using integration by parts, we find a relation between the two results:
$$S_1=-2\pi i\int_{-\infty}^{\infty}dx\space h(x)\frac{d}{dx}f(x)\color{lightgrey}{+2\pi i \space h(x)f(x)\vert^{\infty}_{-\infty}}$$
$$=2\pi i\int_{-\infty}^{\infty}dx\space f(x)\frac{d}{dx}h(x)-2\pi i \space h(x)f(x)\vert^{\infty}_{-\infty}\color{lightgrey}{+2\pi i \space h(x)f(x)\vert^{\infty}_{-\infty}}$$
If the grey terms are correct, this amounts to:
$$S_1=S_2+2\pi i \space h(x)f(x)\vert^{\infty}_{-\infty}$$
If the grey terms are redundant, this amounts to:
$$S_1=S_2-2\pi i \space h(x)f(x)\vert^{\infty}_{-\infty}$$
In any case, these results are different!
Second question:
Seems like we got different results using the two options, based on how we choose to represent the integral over $\omega$: a derivative of a delta function with respect to $x$, or rather, $y$.
What is the source of this contradiction? What is the correct result of the integration?
When in doubt, switch to nascent deltas until near the end to see whether our original problem is even well-defined. (A "contradiction" here would really mean that it isn't.)
For $\epsilon>0$, define $$I_\epsilon:=\int_{\Bbb R}dx f(x)\int_{\Bbb R}dy h(y)\int_{-1/\epsilon}^{1/\epsilon} d\omega\omega\exp i\omega(x-y)$$ and a nascent delta $$\delta_\epsilon(x-y):=\frac{1}{2\pi}\int_{-1/\epsilon}^{1/\epsilon} d\omega\exp i\omega(x-y)$$ so $$I_\epsilon=-2\pi i\int_{\Bbb R}dx f(x)\int_{\Bbb R}dy h(y)\delta_\epsilon^\prime(x-y),$$ where $\prime$ on a function will indicate differentiation with respect to its argument throughout herein. Integration by parts gives $$\frac{I_\epsilon}{2\pi i}=\int_{\Bbb R}dx f^\prime(x)\int_{\Bbb R}dy h(y)\delta_\epsilon(x-y)-\int_{\Bbb R}dyh(y)\left[f(x)\delta_\epsilon(x-y)\right]_{-\infty}^\infty.$$(We'll leave aside, for now, the reasons to expect that the boundary term vanishes.) Similarly, we can get $$\frac{I_\epsilon}{2\pi i}=-\int_{\Bbb R}dx f(x)\int_{\Bbb R}dy h^\prime(y)\delta_\epsilon(x-y)+\int_{\Bbb R}dxf(x)\left[h(y)\delta_\epsilon(x-y)\right]_{-\infty}^\infty.$$ Equating these, $$\int_{\Bbb R^2}dxdy\left(f^\prime(x)h(y)+f(x)h^\prime(y)\right)\delta_\epsilon(x-y)\\=\int_{\Bbb R}dyh(y)\left[f(x)\delta_\epsilon(x-y)\right]_{-\infty}^\infty+\int_{\Bbb R}dxf(x)\left[h(y)\delta_\epsilon(x-y)\right]_{-\infty}^\infty.$$It helps to rewrite the second term on the right-hand side by exchanging the names of the labels $x,\,y$. Since $\delta_\epsilon$ is even, our revised right-hand side is $$\int_{\Bbb R}dy\left\{ h(y)\left[f(x)\delta_\epsilon(x-y)\right]_{-\infty}^\infty+f(y)\left[h(x)\delta_\epsilon(x-y)\right]_{-\infty}^\infty\right\}.$$If we take the distributional limit $\epsilon\to 0^+$, the consistency condition reduces to $$\int_{\Bbb R}dx(f(x)h(x))^\prime=2[f(x)h(x)]_{-\infty}^\infty$$ or equivalently $[f(x)h(x)]_{-\infty}^\infty$. This is necessary for $I$ to exist. Let's compare this with the situation for $I_\epsilon$: as long as $f,\,h$ have such behaviour at $\pm\infty$ as to ensure $$\lim_{x\to\infty}\delta_\epsilon(x-y)=0\implies\lim_{x\to\infty}f(x)h(y)\delta_\epsilon(x-y)=\lim_{x\to\infty}f(y)h(x)\delta_\epsilon(x-y)=0,$$we get $$\int_{\Bbb R^2}dxdy\left(f^\prime(x)h(y)+f(x)h^\prime(y)\right)\delta_\epsilon(x-y)=0,$$which again recovers $[f(x)h(x)]_{-\infty}^\infty=0$ in the distributional limit.
As a final point, note that the original problem's containing $\delta$ means it will be well-posed in particular for Schwartz functions $f,\,h$ (because $\delta$ is a tempered distribution), which unsurprisingly guarantees the desired consistency condition.