Given two functors, $F$ and $G$, between categories $\mathbf{C}$ and $\mathbf{D}$, a natural transformation $\eta$ associates a morphism $\eta_X$ for every $X$ in $\mathbf{C}$. This morphism is between objects in $\mathbf{D}$. From what I read, that is (part of) the definition for a natural transformation.
My understanding of what is and is not a morphism is fuzzy. Is there any requirement that $\eta_X$ be a morphism in $\mathbf{D}$ (i.e. $\eta_X \in \operatorname{hom}(\mathbf{D})$), or is it permitted to be any arbitrary mapping between two objects in $\mathbf{D}$?
As a concrete example, consider a very simple pair of categories, $\mathbf{C}$ and $\mathbf{D}$.
- Category $\mathbf{C}$ has two members, $A$ and $B$, and three morphisms: the two identity morphisms and $x$ where $x$ is from $A$ to $B$.
- Category $\mathbf{D}$ has four members, $A_1$, $A_2$, $B_1$ and $B_2$. It has six morphisms: the four identity morphisms, $x_1$ from $A_1$ to $B_1$ and $x_2$ from $A_2$ to $B_2$. There are obviously two functors from $\mathbf{C}$ to $\mathbf{D}$: $F$ maps $A$ to $A_1$, $B$ to $B_1$ and $x$ to $x_1$. $G$ maps $A$ to $A_2$, $B$ to $B_2$ and $x$ to $x_2$.
Is there a natural transformation between $F$ and $G$? Clearly the transformation would have $\eta_A \colon A_1 \to A_2$ and $\eta_B \colon B_1 \to B_2$ However, neither $\eta_A$ nor $\eta_B$ are in $\operatorname{hom}(\mathbf{D})$. I can't tell if that's intended to be part of the definition or not.
If there is such a natural transformation between $F$ and $G$, is there a common name for the subcategory of natural transformations where $\eta_X$ is in $\mathbf{D}$ for all $X$ in $\mathbf{D}$?
There are no “arbitrary mapping between two objects in $\mathbf{D}$”, and no morphisms between objects of $\mathbf{D}$ except for those in $\operatorname{hom}(\mathbf{D})$. A “morphism in a category $\mathbf{D}$” is simply defined as “an element of $\operatorname{hom}(\mathbf{D})$”.
In other words: we can only talk about morphisms in the context of an ambient category.
To define what a natural transformation is, we need the following data:
A natural transformation $η$ from $F$ to $G$ associates to every object $X$ of $\mathbf{C}$ a morphism $η_X$ from $F(X)$ to $G(X)$ in $\mathbf{D}$. In other words: $η_X$ is by definition required to be an element of the set $\operatorname{Hom}_{\mathbf{D}}(F(X), G(X))$. This means in particular that each $η_X$ is an element of $\operatorname{hom}(\mathbf{D})$.
To get to your specific example: you write that “neither $η_A$ nor $η_B$ are in $\operatorname{hom}(\mathbf{D})$”. However, the problem is not that $η_A$ and $η_B$ exist outside of $\operatorname{hom}(\mathbf{D})$, but that they simply don’t exist.
There is, however, a way in which the idea of “morphisms outside of $\mathbf{D}$” can be made rigorous.
Suppose that $\mathbf{E}$ is a category that contains $\mathbf{D}$ as a subcategory. Let $I$ be the inclusion functor from $\mathbf{D}$ to $\mathbf{E}$.
We may think about the functors $F$ and $G$ not only as functors from $\mathbf{C}$ to $\mathbf{D}$, but as functors from $\mathbf{C}$ to $\mathbf{E}$. This means that we consider the two functors $I ∘ F$ to $I ∘ G$ instead of the original functors $F$ and $G$. We can now study natural transformations between $I ∘ F$ and $I ∘ G$. If $η$ is such a natural transformation from $I ∘ F$ and $I ∘ G$, then its components $η_X$ are morphisms in $\mathbf{E}$, i.e., they are elements of $\operatorname{hom}(\mathbf{E})$.
Now, $\operatorname{hom}(\mathbf{D})$ is a subclass of $\operatorname{hom}(\mathbf{E})$. We can therefore ask whether every component $η_X$ is already an element of $\operatorname{hom}(\mathbf{D})$. In other words: it makes sense to ask whether all $η_X$ are already morphisms in $\mathbf{D}$. If this is the case, then we can restrict $η$ to a natural transformation $\tilde{η}$ between the functors $F$ and $G$. These natural transformations $$ η \colon I ∘ F \Rightarrow I ∘ G \,, \quad \tilde{η} \colon F \Rightarrow G $$ have the same components, in the sense that $η_X = \tilde{η}_X$ for every object $X$ of $\mathbf{C}$. But they are not the same natural transformation, because they are between different functors.
In this situation, there may be more transformations from $I ∘ F$ to $I ∘ G$ than there are natural transformations from $F$ to $G$.
To get back to your specific example: We may define a new category $\mathbf{E}$ that has four objects and eight morphisms. Its objects are $A_1$, $B_1$, $A_2$ and $B_2$. Its morphisms are the four identity morphisms, a morphism $x_1$ from $A_1$ to $B_1$, a morphism $x_2$ from $A_2$ to $B_2$, a morphism $y$ from $A_1$ to $A_2$ and a morphism $z$ from $B_1$ to $B_2$. Composition of morphisms is defined in the only way possible. (This is the category corresponding to the partially ordered set with elements $A_1$, $B_1$, $A_2$, $B_2$ and partial order $≤$ given by $A_1 ≤ B_1$, $A_2 ≤ B_2$, $A_1 ≤ A_2$ and $B_1 ≤ B_2$.) The category $\mathbf{D}$ is a subcategory of this new category $\mathbf{E}$. Let $I$ be the inclusion functor from $\mathbf{D}$ to $\mathbf{E}$.
There exist no natural transformation from $F$ to $G$ because in the category $\mathbf{D}$ there exist no morphism from $A_1$ to $A_2$, as well as no morphism from $B_1$ to $B_2$. But there exists a (unique) natural transformation $η$ from the functor $I ∘ F$ to the functor $I ∘ G$. Its components are given by $η_A = y$ and $η_B = z$. But $η$ is not a natural transformation from $F$ to $G$.