I'm studying fractional Sobolev spaces and, similarly to a previous question of mine, I have some troubles to understand some definitions.
Consider the Bessel potential spaces, defined as $$ H^{s,p}\left(\mathbb{R}\right):=\left\{ u\in L^{p}\left(\mathbb{R}\right):\,\mathcal{F}^{-1}\left[\left(1+\left|\cdot\right|^{2}\right)^{s/2}\mathcal{F}\left[u\right]\right]\in L^{p}\left(\mathbb{R}\right)\right\},$$ where $s>0$, $\mathcal{F}\left[\cdot\right]$ is the Fourier transform and $1<p<+\infty$, equipped with the norm $$\left\Vert \mathcal{F}^{-1}\left[\left(1+\left|\cdot\right|^{2}\right)^{s/2}\mathcal{F}\left[u\right]\right]\right\Vert _{L^{p}\left(\mathbb{R}\right)}.$$ Now, my problem is to understand what is, explicitly, such norm. This is what I did: the Fourier transform of a $L^{p}\left(\mathbb{R}\right)$ function could not be a function, so we need to use the distributional definition of such integral transform. Hence $$\mathcal{F}\left[u\right]=\mathcal{F}\left[u\right]_{\phi}=\int_{\mathbb{R}}u\left(x\right)\mathcal{F}\left[\phi\right]\left(x\right)dx$$ where $\phi(x)$ is a test fuction. The product of a smooth function with a distribution is always defined, since, if $\psi$ is smooth, then $$\left\langle \psi u,\phi\right\rangle =\left\langle u,\psi\phi\right\rangle $$ hence it should be $$\left(1+\left|\cdot\right|^{2}\right)^{s/2}\mathcal{F}\left[u\right]=\int_{\mathbb{R}}u\left(x\right)\left(1+\left|x\right|^{2}\right)^{s/2}\mathcal{F}\left[\phi\right]\left(x\right)dx$$ because $$\mathcal{F}\left[u\right]=\left\langle u,\mathcal{F}\left[\phi\right]\right\rangle $$ and so $$\left(1+\left|\cdot\right|^{2}\right)^{s/2}\mathcal{F}\left[u\right]=\left\langle \left(1+\left|\cdot\right|^{2}\right)^{s/2}u,\mathcal{F}\left[\phi\right]\right\rangle =\left\langle u,\left(1+\left|\cdot\right|^{2}\right)^{s/2}\mathcal{F}\left[\phi\right]\right\rangle. $$ Hence $$\mathcal{F}^{-1}\left[\left(1+\left|\cdot\right|^{2}\right)^{s/2}\mathcal{F}\left[u\right]\right]=\int_{\mathbb{R}}u\left(x\right)\mathcal{F}^{-1}\left[\left(1+\left|y\right|^{2}\right)^{s/2}\mathcal{F}\left[\phi\right]\left(y\right)\right]\left(x\right)dx=:\int_{\mathbb{R}}u\left(x\right)\tilde{v}_{\phi}\left(x\right)dx$$ and finally $$\left\Vert \mathcal{F}^{-1}\left[\left(1+\left|\cdot\right|^{2}\right)^{s/2}\mathcal{F}\left[u\right]\right]\right\Vert _{L^{p}\left(\mathbb{R}\right)}=\sup\left\{ \int_{\mathbb{R}}u\left(x\right)\tilde{v}_{\phi}\left(x\right)dx:\,\left\Vert \phi\right\Vert _{q}\leq1\right\},$$
where $q$ is the conjugate of $p$. I'm really not sure if this ideas are correct, and I would like to know if there are some mistakes.
You are right in saying that $\mathcal F[u]$ might not be a function any more. However, why not work with distributions directly? You know that $\mathcal F[u]$ is defined for any tempered distribution, and the result will be a tempered distribution again. Therefore $(1+|\cdot|^{2})^{s/2}\mathcal{F}[u]$ will be a temp. distribution again, and you may apply the inverse Fourier transform, i.e. $u_s:=\mathcal F^{-1}[(1+|\cdot|^{2})^{s/2}\mathcal{F}[u]]$ is well-defined and a tempered distribution.
Finally, $u\in H^{s,p}$ iff $u_s\in L^p$.
Comment: The factor $(1+|\cdot|^2)^{s/2}$ in Fourier-space can be interpreted as a fractional derivative of the original function. To see this, recall that $\mathcal F[u'](\xi) = \mathrm{i}\xi \mathcal F[u](\xi)$. This illustrates the general fact that the asymptotic decay of $\mathcal F[u](\xi)$ as $|\xi|\to\pm\infty$ is closely linked to the smoothness of $u$. In particular, you could define $H^1(\mathbb R)$ as the set of all $L^2$-functions for which $$\mathcal F^{-1}\big[(1+\xi^2)^{1/2}\mathcal F[u](\xi)\big]\in L^2.$$ It really says that whenever the inverse transform (which always exists as a tempered distribution) also happens to be an $L^2$-function, the function is in $H^1$.