Consider the Fourier transform of the Heaviside function $H(x)$: $$ \int_{-\infty}^{\infty} H(x) e^{- i p x} dx = \int_{0}^{\infty} e^{- i p x} dx = - \frac{i}{p} + \pi \delta(p) $$ By shifting the integration variable by some amount $y>0$ one eventually can derive the result $$ \int_{y}^{\infty} e^{- i p x} dx = - \frac{i}{p} e^{- i p y} + \pi \delta(p) $$ where I've used $\delta(p)f(p) = \delta(p)f(0)$.
Finally, if I differentiate this expression with respect to $p$, I eventually find $$ \int_{y}^{\infty} p e^{- i p x} dx = \left( - \frac{1}{p^2} - \frac{i y}{p} \right) e^{- i p y} + i \pi \delta'(p) $$
I am confused about this result (which I think is correct), because as one takes the limit $y \to \infty$ the function on the RHS diverges. This is strange because, naively the LHS seems like it should vanish, since the integration region approaches a set of width $0$ as we take $y \to \infty$.
Why is this happening? Is there a simple way to understand this seemingly contradictory behaviour?
If the object written $\int_y^\infty xe^{-ipx}\,dx$ is interpreted as an integral, it diverges for all $p\in\mathbb{R}$.
However, we can interpret the object as a distribution. Let $f(x)=xH(x-y)$. Then we have in distribution
$$\begin{align} \int_y^\infty xe^{-ipx}\,dx&=\mathscr{F}\{f\}(p;y)\\\\ &=i\frac{d}{dp}\left(-\frac ip e^{-ipy}+\pi \delta(p)\right) \end{align}$$
We will now show that in distribution,
$$\lim_{y\to\infty}\mathscr{F}\{f\}(p;y)=\lim_{y\to\infty}i\frac{d}{dp}\left(-\frac ip e^{-ipy}+\pi \delta(p)\right)=0\tag1$$
Let $\phi\in \mathbb{S}$ (i.e., $\phi$ is a Schwartz space function). Then, we clearly have
$$\begin{align} \lim_{y\to \infty} \langle \mathscr{F}\{f\},\phi\rangle &=\lim_{y\to \infty} \langle f,\mathscr{F}\{\phi\}\rangle \\\\ &=\lim_{y\to \infty} \int_{y}^\infty x\mathscr{F}\{\phi\}(x)\,dx\\\\ &=0\tag2 \end{align}$$
where $(2)$ is a consequence of the Cauchy Criterion for the convergent improper integral $\int_{y}^\infty x\mathscr{F}\{\phi\}(x)\,dx$.
Let $\psi=\frac 1p e^{-ipy}+i\pi \delta(p)$ so that $\psi'=-\left(\frac1{p^2}+\frac{iy}{p}\right)e^{-ipy}+i\pi \delta'(p)$. In distribution, we have
$$\begin{align} \lim_{y\to \infty} \langle \mathscr{F}\{f\},\phi\rangle &=\lim_{y\to \infty} \langle \psi',\phi\rangle \\\\ &=-\lim_{y\to \infty} \langle \psi, \phi'\rangle\\\\ &=-\lim_{y\to \infty}\left(\int_{-\infty}^\infty \frac{\phi'(p)}{p}e^{-ipy}\,dp+i\pi \phi'(0)\right)\\\\ &=-\lim_{y\to \infty}\int_{-\infty}^\infty \frac{\phi'(p)-\phi'(0)}{p}e^{-ipy}\,dp\\\\ &=0\tag3 \end{align}$$
where we relied on the Riemann-Lebesgue Lemma to arrive at $(3)$.
Putting $(2)$ and $(3)$ together yields $(1)$ as was to be shown!