The main refference: Prove that sheaf hom is a sheaf.
Without even knowing about the codomain of
$$\mathcal Hom(\mathscr F,\mathscr G):Open(X)^{op}\to (???)$$
Defining $\mathcal Hom(\mathscr F,\mathscr G)(U):=Nat(\mathscr F|_U,\mathscr G|_U)$ and proper restriction maps for $U\subset V$
$$\mathcal Hom(\mathscr F,\mathscr G)(V)\xrightarrow{\bar{res}_{V,U}} \mathcal Hom(\mathscr F,\mathscr G)(U)$$
One can show it is "a" sheaf on $X$. But I want to understand this $(???)$ the codomain of the functor $\mathcal Hom(\mathscr F,\mathscr G)$ so that I then can understand what happens if I change or take out $\mathscr F,\mathscr G$.
Objects of $(???)$ consist sets/classes of natural transformations between fixed restricted sheaves. But then I have seen Morphisms in the category of natural transformations?
So what is the proper way to understand $(???)$
Thorgott's comment is correct, but I think looking at a few concrete examples may clarify your confusion.
In the case that $\mathscr{F,G}$ are sheaves of sets on a space $X$ (or any site, if you prefer/require such generality), then for each $U\subseteq X$ open, $\mathrm{Hom}(\mathscr{F}\vert_U,\mathscr{G}\vert_U)$ is a set, and so $\mathcal{H}\mathrm{om}(\mathscr{F},\mathscr{G})$ can be well-regarded as a sheaf of sets. The point is, the hom-sets in the category $\mathbf{Set}$ are again sets.
If $\mathscr{F,G}$ are sheaves of abelian groups, then the collection of sheaf homomorphisms $\mathrm{Hom}(\mathscr{F}\vert_U,\mathscr{G}\vert_U)$ will not just be a set, but will be equipped with the structure of an abelian group. So we can think of this "sheaf-hom" as a sheaf of abelian groups. This is because the category of sheaves of abelian groups is an abelian category, so its hom-sets are abelian groups.
This can be resolved in general by the theory of enriched categories, but I think any further generality will probably serve to confuse rather than illuminate. So I will just say this: whatever category your hom-sets belong to is the right target for the "sheaf-hom" just like in the above examples.