Consider the Hom-Set Functor $H : C^{op} \times C \rightarrow \mathbf{Set}$.
If we let $c_1, c_2$ be categories (and assume, as is tradition, that all hom-sets are actually sets), then we have that $H(c_1, c_2)$ is a hom-set in $\mathbf{Set}$.
Question: Since $h \in H(c_1, c_2)$ is a function from $Ob(c_1)$ to $Ob(c_2)$, doesn't that imply that $Ob(c_1)$ and $Ob(c_2)$ are themselves sets? But how can we assume that?
Let me try to lay things out straight.
Given a category $\mathbf{C}$ with small hom-sets, there's a functor $$\mathrm{Hom} : \mathbf{C}^{op} \times \mathbf{C} \rightarrow \mathbf{Set}.$$
Given objects $X$ and $Y$ in $\mathbf{C}$, $\mathrm{Hom}(X,Y)$ is the set of all morphisms $X \rightarrow Y$.
The elements of $\mathrm{Hom}(X,Y)$ aren't necessarily functions.
Supposing $\mathbf{C} = \mathbf{Set}$, then $X$ and $Y$ are sets and $\mathrm{Hom}(X,Y)$ does indeed consist of functions. But the domain of $f \in \mathrm{Hom}(X,Y)$ is $X$, and not $\mathrm{Ob}(c_1)$ for some auxilliary category $c_1$.
Does that help?