In the first pages of "category theory for the working mathematician" Saunders claims that category can be introduced, without set theory, as objects and arrows without some "operations" satisfying some properties which are called axioms. (He calls these metacategories.)
I like that approach, but either I just try to understand things intuitively, either if someone claims that everything is well-defined then I want to see it. What I don't see is the definition of "operation" which "assigns an object to an arrow", ex: the "domain" operation assigns to an arrow an object called domain, same for codomain, composition etc...
More precisely, if we get rid of set theory, then what is a function, what is a pair, what is something that belongs to something?
Another question: is the need to start with other axioms then those of set theory only related to the notion of "class" of objects or is it problematic in other places?
I think the situation is that the primitive notions being introduced are "objects," and "arrows," and that the assigment of "domain" and "codomain," "identity," and "composition" are also primitive functions. Along with the axioms about how these interact, this constitutes a suitable environment to do category theory.
There's nothing wrong with having functions that are primitive notions. Remember that $\in$ itself is a primitive relation, and it doesn't (can't!) have any set theoretic justification for its relation-ness.
Does the edition of Categories... you are using have the appendix on these foundations? He does a pretty good job there of explaining how they do their job in the absence of regular set theory.