How to prove that something is a set?

Question

In Rotman's "An Introduction to Algebraic Topology", he insists that for any objects $A, B$ in a category $C$, $Hom(A, B)$ has to be a set. But in practice, how do you actually prove that something is a set? I am totally new to this categorical stuff and really confused. The only explanation provided in the book is that a set is a class small enough to have a cardinality. However, none of those words make sense to me.

Answer 1

There are many things going on in your question, and I'll try to address what I think are the main parts.

To begin with, the definition of category you have encountered isn't the only possible one, but for everything you will probably care about they are roughly the same. It's very common for books to state briefly what kind of foundations they assume, and then they never mention them again.

Second, the thing that seems to me as the main question, how one knows that the homsets of a category are sets, seems to me to be more about foundations and set theory: if you haven't encountered the distinction between proper classes and sets before, the problem that they are made to solve is essentially that of things being self-containing, like "the set of all sets", which isn't an allowed construction.

One possible way to deal with it is to define sets quite recursively by postulating the existence of an empty set and then only allow certain operations and ways to form new ones. Taking the collection of all functions between two sets gives a set, for instance, so for most of the categories you care about, you don't need to prove that anything is a set. Overall, it's not very often that these considerations actually come up in practice, unless you're trying to consider things like the category of categories and so on. In those situations you need to be careful, and actually consider which flavour of foundational approaches that the book is supposed to use.