I cannot get this identity by using the condition:
$\frac{1}{x^{2}}=-\partial(\frac{1}{x})$, and the integrands are defined by continuity at $x=0$.
My reasoning goes as $$\langle -\partial\frac{1}{x},\phi\rangle=\langle \frac{1}{x},\partial \phi\rangle=\int^{\infty}_{-\infty}\frac{1}{x^{2}}\phi dx$$ and we can break it into $$\lim_{\epsilon\rightarrow 0}\int^{\infty}_{\epsilon}\frac{1}{x^{2}}\phi dx+\int^{-\epsilon}_{-\infty}\frac{1}{x^{2}}\phi dx+\int^{\epsilon}_{-\epsilon}\frac{1}{x^{2}}\phi dx$$ The first and second term adds up to $$\int^{\infty}_{\epsilon} \frac{\phi(x)+\phi(-x)}{x^{2}}dx$$ while the third term $$\int^{\epsilon}_{-\epsilon}\frac{1}{x^{2}}\phi(x) dx=\int^{\epsilon}_{0}\frac{\phi(x)+\phi(-x)}{x^{2}}$$ approximates $$2\int^{\epsilon}_{0}\frac{\phi(0)}{x^{2}}$$
But this last step is not really justified; and even if it works the result looks remarkably different from the one I wanted. So I decided to ask.
Let $\phi$ be a smooth test function. Since $\big\langle \frac{1}{x^2}, \phi \big\rangle = \big\langle -\partial \left( \frac{1}{x} \right), \phi \big\rangle = \big\langle \frac{1}{x}, \partial \phi \big\rangle$, it suffices to show that
$$ \left< \frac{1}{x}, \partial \phi \right> = PV \int_{-\infty}^{\infty} \frac{\partial \phi (x)}{x} \; dx = \int_{0}^{\infty} \frac{\phi(x) + \phi(-x) - 2\phi(0)}{x^2} \; dx.$$
But this follows from
$$ \begin{align*} PV \int_{-\infty}^{\infty} \frac{\partial \phi (x)}{x} \; dx &= \lim_{\epsilon\downarrow 0} \left( \int_{-\infty}^{-\epsilon} \frac{\partial \phi (x)}{x} \; dx + \int_{\epsilon}^{\infty} \frac{\partial \phi (x)}{x} \; dx \right) \\ &= \lim_{\epsilon\downarrow 0} \int_{\epsilon}^{\infty} \frac{\partial \phi (x) - \partial \phi(-x)}{x} \; dx \\ &= \lim_{\epsilon\downarrow 0} \left( \left. \frac{\phi (x) + \phi(-x)}{x} \right|_{\epsilon}^{\infty} + \int_{\epsilon}^{\infty} \frac{\phi (x) + \phi(-x)}{x^2} \; dx \right) \\ &= \lim_{\epsilon\downarrow 0} \left( - \frac{\phi (\epsilon) + \phi(-\epsilon)}{\epsilon} + \int_{\epsilon}^{\infty} \frac{\phi (x) + \phi(-x)}{x^2} \; dx \right) \\ &= \lim_{\epsilon\downarrow 0} \left( \frac{2\phi(0) - \phi (\epsilon) - \phi(-\epsilon)}{\epsilon} + \int_{\epsilon}^{\infty} \frac{\phi (x) + \phi(-x) - 2\phi(0)}{x^2} \; dx \right) \\ &= \int_{0}^{\infty} \frac{\phi (x) + \phi(-x) - 2\phi(0)}{x^2} \; dx, \end{align*}$$
where we have used the fact that
$$ \lim_{\epsilon\downarrow 0} \frac{\phi (\epsilon) + \phi(-\epsilon) - 2\phi(0)}{\epsilon} = 0.$$