I'm trying to digest the following statements, which are taken from §2 of Carlo Morpurgo's paper Sharp inequalities for functional integrals and traces of conformally invariant operators.
For $0< \alpha < n$, up to a positive constant multiple, one can define the fractional Laplacian of order $n$ on the Schwartz class $\mathcal S(\mathbb R^n)$ as \begin{equation*} \mathcal F((-\Delta)^{\alpha/2}\varphi)(\xi) = |\xi|^\alpha \mathcal F\varphi(\xi), \qquad \text{ for }\varphi\in \mathcal S(\mathbb R^n), \end{equation*} where $\mathcal F$ is the Fourier transform. This operator can be extended by duality to the space of tempered distributions $\mathcal S'(\mathbb R^n)$.
If $p\geq 1$ and $f\in L^p(\mathbb R^n)$ is given, the equation $(-\Delta)^{\alpha/2} u = f$ has a unique solution in $L^q(\mathbb R^n)$, where $\frac 1 q = \frac 1 p - \frac \alpha n$. Up to constant multiple, this solution is given explicitly by convolution with the Riesz potential: \begin{equation*} u(x) = I_\alpha (f):=\int_{\mathbb R^n}\frac{f(y)}{|x - y|^{n - \alpha}}\; \mathrm d y. \end{equation*}
My first question is in regards to the extension of $(-\Delta)^{\alpha/2}$ to $\mathcal S'(\mathbb R^n)$ via duality. Heuristically, for $v\in \mathcal S'(\mathbb R^n)$ one would like to define \begin{equation*} \langle (-\Delta)^{\alpha/2}v, \varphi\rangle := \langle v, (-\Delta)^{\alpha/2}\varphi\rangle = \langle \mathcal F v , |\xi|^\alpha \mathcal F\varphi\rangle \end{equation*} whenever $\varphi \in \mathcal S(\mathbb R^n)$. The problem with this heuristic is that $\mathcal F v$, being a tempered distribution, acts on functions in $\mathcal S(\mathbb R^n)$. However, $|\xi|^\alpha \mathcal F\varphi\notin \mathcal S(\mathbb R^n)$ due to the lack of differentiability at $\xi = 0$. What is the correct interpretation of the action of $\mathcal F v$ on $|\xi|^\alpha \mathcal F\varphi$?
My second question is in regard to the statement that $I_\alpha f$ satisfies $(-\Delta )^{\alpha/2} u = f$ whenever $f\in L^p$ (of particular interest is $p = 2n/(n + \alpha)$ and $q = 2n/(n - \alpha)$, but that doesn't matter for the question at hand). The Hardy-Littlewood-Sobolev inequality guarantees that $I_\alpha f\in L^q$, so $((-\Delta)^{\alpha/2}\circ I_\alpha )(f)$ should be interpreted in the distributional sense. In particular, for $\varphi\in \mathcal S(\mathbb R^n)$ I would like to perform the following computation (up to constant multiple): \begin{eqnarray*} \langle (-\Delta)^{\alpha/2}\circ I_\alpha f, \varphi\rangle & = & \langle I_\alpha f, (-\Delta)^{\alpha/2}\varphi\rangle\\ & = & \langle \mathcal F(I_\alpha f), \mathcal F((-\Delta)^{\alpha/2} \varphi)\rangle\\ & = & \langle |\xi|^{-\alpha}\mathcal F f, |\xi|^\alpha \mathcal F\varphi\rangle\\ & = & \langle\mathcal F f, \mathcal F\varphi\rangle\\ & = & \langle f, \varphi\rangle. \end{eqnarray*} The main problem is the interpretation of the expression $\langle \mathcal F(I_\alpha f), \mathcal F((-\Delta)^{\alpha/2} \varphi)\rangle$. Since $I_\alpha f\in L^q$, we can interpret $\mathcal F(I_\alpha f)$ as a tempered distribution. However, with this interpretation, $\mathcal F (I_\alpha f)$ can not act on $\mathcal F((-\Delta)^{\alpha/2}f)$ since the latter of these two quantities is not in the Schwartz class. What is the correct interpretation of the action of $\mathcal F(I_\alpha f)$ on $\mathcal F((-\Delta)^{\alpha/2}\varphi)$?
To the first question, I don't think there is a way to extend the fractional Laplacian to all tempered distributions. The difficulty is exactly as you stated, that multiplication by $|\xi|^{\alpha}$ doesn't preserve the Schwartz class $\mathcal{S}$. That said, if the distribution is such that its Fourier transform agrees with a measure or function in a neighborhood of the origin, we can make things work by keeping the multiplication by $|\xi|^{\alpha}$ on the distribution rather than the Schwartz function, at least in that neighborhood.
To the second question, there's a slightly different way of looking at things that avoids the difficulty in the calculation. To begin, even though the fractional Laplacian $(-\Delta)^{\alpha/2}$ doesn't preserve the Schwartz space, it does map this space into $L^r$ for $1 \leq r \leq \infty$. (By looking at the inverse Fourier transform of $|\xi|^{\alpha} \mathcal{F} \varphi(\xi)$ and using integration by parts, we can obtain generic decay at infinity on the order of $|x|^{-n - \alpha}$.) Then for $f \in L^p$, and letting $u = I_{\alpha} f \in L^q$ (with $1/p = 1/q - \alpha/n$), one (weak type) sense in which $(-\Delta)^{\alpha/2} u = f$ is that, for $\varphi \in \mathcal{S}$, $$ \int u(x) \overline{(-\Delta)^{\alpha/2} \varphi(x)} \, dx = \int f(x) \overline{\varphi(x)} \, dx. $$ Heuristically the same thing is going on as in the not-quite-formally-correct distributional calculation, of course. Perhaps the key thing to keep in mind is the basic fact that distributions given by $L^p$ functions act via integration, and so the integration point of view can help with some of the formalities.