Is it true that for every unitary extension $U' \supset U$
$\psi \in \{ {\frak Dom}(U)\}^\perp \Longleftrightarrow U'\psi \in \{ U'({\frak Dom} (U))\}^\perp$
?
My question is: how can I be sure that $\{ {\frak Dom}(U)\}^\perp \subset \{{\frak Dom }(U')\}$? And why does $U' \psi$ belong to $\{ U'({\frak Dom} (U))\}^\perp$ and not simply to $\{{\frak Ran }(U')\}$?
Could you prove it?
It is true that ${\frak Dom}(U)^\perp \subset {\frak Dom }(U')$ if we assume that orthogonal complement in ${\frak Dom}(U)^\perp$ is taken relative to the same space on which the extension $U'$ is defined. More precisely, suppose $B$ is a vector space and $U: A\to A$ is a unitary map defined on a subspace $A = {\frak Dom}(U)$ of $B$. Consider an extension $U':B\to B$ of $U$ defined on the whole space $B$. In this case, we have
$${\frak Dom}(U)^\perp = A^\perp \subset B = {\frak Dom}(U')$$
by construction.
Affirmative answer to the second question follows from the fact that the inverse of a unitary map is its Hermitian adjoint. Let $\psi \in {\frak Dom}(U)^\perp$. To prove that $U'\psi \in (U'{\frak Dom}(U))^\perp$ we need to show that $\langle U'\psi, \phi\rangle = 0$ for every $\phi \in U'{\frak Dom}(U)$. However,
$$ \langle U'\psi, \phi\rangle = \langle \psi, U'^{-1}\phi\rangle = 0 $$
where the last equality follows from the facts that $U'^{-1}\phi \in {\frak Dom}(U)$ and $\psi \in {\frak Dom}(U)^\perp$.