I want to prove that convolution of a function with itself smooths the function in some sense.
Given a function $f:\mathbb{R}^n \to \mathbb{R}$, the roughness $R = \int \Delta f $ where we integrate the Laplacian over all values of $f$.
Can we show that $R(f * f) < R(f)$ where $*$ is convolution. If not, is there another measure of smoothness we could use.
I'm not sure the "roughness" you define is a good measure of smoothness. For instance, if $n=1$, and $f$ is twice differentiable, then $$\int_{\mathbb R} \Delta f=\lim_{+\infty}f^\prime-\lim_{-\infty}f^\prime$$ when those limits exist. It only depends on the behavior of $f^\prime$ at the infinities.
Instead, you can look at something like a Sobolev norm $$\|f\|_S^2=\int |f|^2 + \int |\Delta f|^2$$ You see, if $f$ is not super smooth, then the Laplacian will have a lot of energy, and the norm will be high. So small Sobolev norm means smooth function.
Take $f \in L^1(\mathbb R^n)\cap L^2(\mathbb R^n)$. Then $f$ has a Fourier transform. Thanks to Parseval's theorem, the Sobolev norm can be expressed in the Fourier domain: $$\|f\|_S^2=\int_{\mathbb R^n}(1+\|\omega\|^2)|\hat{f}(\omega)|^2d\omega$$ Also, remember that the Fourier transform maps convolutions to products, so $$\|f*f\|_S^2=\int_{\mathbb R^n}(1+\|\omega\|^2)|\hat{f}(\omega)|^4d\omega$$ Now because $f$ is integrable, its Fourier transform is bounded by its $L^1$ norm $\|f\|_1$ (easy for you to check). So $$\|f*f\|_S^2\leq\|f\|_{1}^2\int_{\mathbb R^n}(1+\|\omega\|^2)|\hat{f}(\omega)|^2d\omega$$ In other words $$\|f*f\|_S\leq \|f\|_1\cdot\|f\|_S$$ Here $\|f\|_1$ acts as a normalization factor. That's necessary because $\|f*f\|_S$ is homogeneous of degree 2, while $\|f\|_S$ is of degree 1. So to make it easier to see the effect of the convolution, just focus on functions that are normalized such that $\|f\|_1=1$, then $$\boxed{\|f*f\|_S\leq \|f\|_S}$$ Convolving decreases the Sobolev norm, that is, the resulting function is smoother.