Let $\Delta$ be the Laplace operator $$ \Delta f = \sum_{i=1}^d \frac{\partial^2 f}{\partial x^2_i}$$ with $Dom(\Lambda) = H^1_0(\mathcal{O}) \cap H^2(\mathcal{O})$ where $\mathcal{O}\subset\mathbb{R}^d$ is a bounded domain with a smooth boundary.
I'm studying from this book [1] and they use the square root of the Laplace operator denoted as $(-\Lambda)^{1/2}$ with domain $Dom((-\Lambda)^{1/2})$ without specifying how does this operator and its domain look like.
Can you either explain me what these symbols denote or recommend me a different publication where I can learn more?
[1] Ruth F Curtain and Hans Zwart. An introduction to innite-dimensional linear systems theory. 1995
It is a standard tool of PDEs and functional analysis known as "functional calculus". In the case of the Dirichlet Laplacian on a bounded domain, take a basis $\{\phi_n\ :\ n\ge 1\}$ of normalized eigenfunctions of $(-\Delta, H^2\cap H^1_0(\mathcal{O}))$ corresponding to the eigenvalues $\{\lambda_n\ :\ n\ge 1\}$. If $u\in H^2\cap H^1_0$, then one can compute its Laplacian via the expansion $$ u=\sum_{n\ge 1} c_n\phi_n, $$ obtaining $$ -\Delta u=\sum_{n\ge 1} c_n \lambda_n \phi_n.$$ Therefore one can define "functions" of $-\Delta$ by the formula $$ f(-\Delta) u=\sum_{n\ge 1} c_n f(\lambda_n)\phi_n,$$ provided that $$ \sum_{n\ge 1}|c_n|^2 |f(\lambda_n)|^2<\infty.$$ This last condition defines the domain of the operator $f(-\Delta)$. This operator is self-adjoint if $f$ is real valued and its eigenvalues are, predictably,
$$ \{f(\lambda_n)\ :\ \lambda_n\ \text{eigenvalue of }-\Delta\}.$$
Taking $f(x)=\sqrt{x}$ one obtains the square root of the Laplacian.
A nice introductory book on those things is the first volume of Zeidler's "Applied functional analysis". For more information, the standard reference is Reed & Simon's four-volume books "Methods of modern mathematical physics".