Let $S$ be a noetherian scheme. Grothendieck [1] constructed for any projective $S$-scheme $X$ the Hilbert scheme $\operatorname{Hilb}_{X/S}$, which represents the functor $$T \mapsto \underline{\operatorname{Hilb}}_{X/S}(T) = \{Y \hookrightarrow X \times_S T \,|\, \text{family of closed subschemes, flat over } T\}.$$ Now if $X, Y$ are both projective over $S$, then the functor $$T \mapsto \underline{\operatorname{Hom}}(X, Y)(T) = \{X_T \to Y_T\}$$ can be embedded into $\underline{\operatorname{Hilb}}_{X \times_S Y/S}(T)$, by sending a morphism $f: X_T \to Y_T$ to its graph $\Gamma_f \subset X_T \times_T Y_T$. This identifies $\underline{\operatorname{Hom}}(X,Y)(T)$ with the subset of schemes $\Gamma \subset X_T \times_T Y_T$, such that the restricted projection $\Gamma \to X$ is an isomorphism. Grothendieck claims that this defines an open subscheme $$\operatorname{Hom}(X,Y) \subset \operatorname{Hilb}_{X \times Y/S}.$$ How can I see that this is the case?
Similarly, Grothendieck claims that $\operatorname{Isom}(X,Y) \subset \operatorname{Hom}(X,Y)$ is an open subset. How can I see this?
[1] Grothendieck, Techniques de construction et théorèmes d'existence en géométrie algébrique IV: les schémas des Hilbert; Séminaire Bourbaki
By definition of the Hilbert scheme, we have a "universal family", i.e. a closed subscheme $i : \mathscr{Z} \to \mathrm{Hilb}_{X\times_S Y/S} \times_S (X\times_S Y)$ such that the composite with the projection $\mathrm{Hilb}_{X\times_S Y/S} \times_S (X\times_S Y) \to \mathrm{Hilb}_{X\times_S Y/S}$ is flat.
Let $\alpha : \mathscr{Z} \to \mathrm{Hilb}_{X\times_S Y/S} \times_S X$ be the composite of $i$ with the projection onto the first two factors. Using Stacks 05XD, we have an open subscheme $\mathscr{U} \subseteq \mathrm{Hilb}_{X\times_S Y/S}$ such that any map $f : W \to \mathrm{Hilb}_{X\times_S Y/S}$ of $S$-schemes factors through $\mathscr{U}$ if and only if the basechange $\alpha_W : \mathscr{Z}_W \to (\mathrm{Hilb}_{X\times_S Y/S} \times_S X)_W$ is an isomorphism. Unwinding definitions, we find $\mathscr{U}$ represents exactly the subfunctor $\underline{\mathrm{Hom}}(X,Y)$ you described.