This is a very basic question about definitions, but I haven't been able to find the answer to it online. If we let $\mathscr{A}$ be an Abelian category, then for any object $A\in\mathscr{A}$, we can define the Hom functor $\mathrm{Hom}_\mathscr{A}(A,-)$.
What category does this take values in? I know that it can take values in $\mathbf{Ab}$, but in the case where $\mathscr{A}=R\hbox{-}\mathbf{Mod}$, the functor seems to take values in $\mathscr{A}$. Which convention holds in general?
On
In a $V$-enriched category, the natural "hom object" of natural transformations between functors $F$ and $G$ is the end $\int V(F,G)$. As a special limit in $V$, it is an object of $V$.
In the Ab-enriched case, modules are functors into Ab, and so hom-objects between them (that is, module homeomorphisms) also form an abelian group.
They also admit a module structure over the center of the domain.
By definition, it takes values in $Ab$.
The special thing about $\mathscr{A}=_RMod$ is that it is an enriched category, enriched over itself, see this wonderful page or Kelly's book.