I'm reading Pseudodifferential Operators by M. E. Taylor, where the author talks about $H^s(\Omega)$ for $s\in\mathbb{R}$ and $\Omega\subset\mathbb{R}^n$ an open set (for example, in the statement of Gårding's inequality) without ever defining it. Indeed, he has only defined such Sobolev spaces for $\mathbb{R}^n$ and compact manifolds. In both of these cases, one has an $s$-order pseudodifferential operator $\Lambda^s$ (with principal symbol $\langle\xi\rangle^s$) which induces an isomorphism $H^s\to L^2$. This could be taken as the definition of $H^s$. However, I don't know how to do the same for general open sets in Euclidean space. Some thoughts:
- On p.51, the author remarks that this is done by altering $\Lambda$ so that it is properly supported. However, I'm unsure what he means by this.
- Perhaps one could use functional calculus on the standard Laplacian $\Delta$. There are several problems with this approach: (a) I would need $\Delta^{s/2}$ to be defined on the space of distributions (so that the definition would be like: a distribution $u$ belongs to $H^s$ if $\Delta^{s/2}u\in L^2$), but functional calculus only defines it on a subspace of $L^2$. (b) Is $\Delta^{s/2}$ really a pseudodifferential operator with the correct symbol?
So what is the correct definition in this context? Any help will be appreciated!
For a general open subset $\Omega$ (without regularity assumptions on its boundary), the Sobolev-spaces $H^s(\Omega)$ are first defined for $s\in \mathbb{N}$ (in the obvious way: derivatives up to order $s$ shall be in $L^2$) and for general $s\in \mathbb{R}$ via interpolation/duality.
However, if $\partial \Omega$ is sufficiently regular there is an easier way: Let's assume for simplicity that $\partial \Omega \in C^\infty$, then one typically defines $H^s(\Omega)$ as the space of distributions on $\Omega$ that admit an extension to $\mathbb{R}^d$ that lies in $H^s(\Omega)$. Equivalently $H^s(\Omega)=rH^s(\mathbb{R}^d)\subset\mathcal{D}'(\Omega)$, where $r:\mathcal{D}'(\mathbb{R}^d)\rightarrow \mathcal{D}'(\Omega)$ is the restriction operator. This yields the same spaces as in the first paragraph.
As a reference on these things I can recommend Taylor's PDE book, which has a whole chapter on various definitions of Sobolev spaces. (Also for $\mathbb{R}^d$ being replaced by a closed manifold).
Now, regarding the comment on properly supported $\psi$do's $\Lambda^s$ you can consider Lemma 7.1 in Shubin's $\psi$do book. Indeed, this states that on an arbitrary manifold $X$ (in particular you could take $X=\Omega$) that there exists a scale of properly supported operators $\Lambda^s\in \Psi^s_{\mathrm{cl}}(X)$ (subscript denoting classicality) with positive principal symbols. Shubin then defines local Sobolev spaces by $H^s_\mathrm{loc}(X)=\{u\in \mathcal{D}'(X): \Lambda^su\in L^2_{\mathrm{loc}}(X)\}$ and proves this to be equivalent with some other definitions.
The point is, that for a general (non-compact) manifold this is as good as it gets: There is no notion of $H^s(X)$ without specifying the behaviour of its functions at infinity. If $X$ happens to be an open subset of $\mathbb{R}^d$ or a closed manifold, the behaviour at infinity (or rather at the boundary) is specified by requiring functions to be extendible across $\partial X$ and we are in the setting of the first few paragraphs.
What if $X$ has a Riemannian metric $g$? I suppose that in this case one could define $H^s(X,g)$ for $s\in \mathbb{N}$ by requiring its functions to satisfy $X_1\dots X_k f \in L^2(M,g)$ for any vector fields $X_1,\dots,X_k$ $(0\le k \le s)$ which satisfy $\vert X_i \vert_g\in L^\infty(X)$. For non-integer $s$ then via interpolation\duality.
If $(X,g)$ happens to be complete (like $\mathbb{R}^d$), then Gaffney showed that the Laplacian $1+\Delta_g$ has a unique self-adjoint realisation in $L^2(X,g)$ and I suppose one could call its domain $\tilde H^2(X,g)$. The same is true for its powers and thus we can define $\tilde H^s(X,g)$ for $s\in 2\mathbb{N}$ and extend to general $s$ by interpolation/duality. I would not be surprised (but have not checked it), if indeed $H^s(X,g)=\tilde H^s(X,g)$ in that case.
You were interested in whether you can define Sobolev spaces on $\Omega$ via powers of the Laplacian. It makes more sense to take powers of $P=1+\Delta$ (in analogy with $\mathbb{R}^d$) and indeed there is a nice theory that tells you that this is possible, at least if you are on a closed manifold. So suppose that $\Omega$ lives inside a closed Riemannian manifold $(M,g)$ (and $\partial \Omega \in C^\infty)$, then $P^z$ is defined for all $z\in \mathbb{C}$ and is a classical $\psi$do of order $\mathrm{Re}(z)$ with the obvious algebraic properties. (This is due to Seeley, but you can find a nice account on it in Shubin's book).
Now you might want to define $H^s(\Omega)=\{f:P^s f\in L^2(\Omega,g)\}$ and at least for $s\in \mathbb{N}$ this gives the same as defined in the beginning, i.e. $H^s(\Omega) = r H^s(M)$. A sufficient criterion for the two spaces to agree is that $P^s$ satisfies the so called transmission condition at $\partial \Omega$: This is Definition 18.2.13 in Hörmander and says that $rP^se_0(C^\infty(\bar \Omega)) \subset C^\infty(\bar \Omega)$, where $e_0$ denotes extension by zero. Now for positive integers-powers $P^s$ is a differential operator and clearly satisfies the condition. For non-integer powers this might fail, as is mentioned at the beginning of page 184 here. This is all I can say about it at the moment.