We know that every locally integrable functions on $\mathbb R^n$ defines naturally a distribution. Moreover, every distribution is differentiable.
I think the following is right:
The fundamental solution to the Laplacian on $\mathbb R^n$ is locally integrable, hence defines a distribution. For example, in $n=3$, we have: $$ \phi(x)=C_3|x|^{-1} $$ And $$ \frac{\partial^2 \phi}{\partial x_j^2}=C_3'[-|x|^{-3}+3x_j^2|x|^{-5}], $$ which is still a distribution but fails to be locally integrable. Am I right?
A bit of computation shows that, for any test function $\varphi \in C_c^{\infty}(\mathbb{R}^3)$ we have
$$ \int_{\mathbb{R}^3} \frac{1}{|x|} \frac{\partial^2 \varphi}{\partial x_j^2}(x) \, dx = -\frac{4\pi}{3}\varphi(0) + \lim_{\epsilon \downarrow 0} \int_{\mathbb{R}^3\setminus B_{\epsilon}(0)} \varphi(x) \frac{\partial^2}{\partial x_j^2}\frac{1}{|x|} \, dx. $$
If you want to replicate this result, you can utilize the divergence theorem to compute both
$$ \int_{\mathbb{R}^3\setminus B_{\epsilon}(0)} \operatorname{div}\left( \frac{1}{|x|} \, \mathrm{e}_j \frac{\partial \varphi}{\partial x_j} (x) \right) \, dx \qquad \text{and} \qquad \int_{\mathbb{R}^3\setminus B_{\epsilon}(0)} \operatorname{div}\left( \varphi (x) \, \mathrm{e}_j \frac{\partial}{\partial x_j} \frac{1}{|x|} \right) \, dx. $$
and then let $\epsilon \downarrow 0$.
Here, the failure of local integrability of $\partial_{jj} \frac{1}{|x|}$ near $x = 0$ is reflected in the emergence of the extra Dirac delta term $-\frac{4\pi}{3}\delta$. This is also where the distributional derivative departs from the ordinary differentiation.