In chapter 4 of Handbook of Categorical Algebra, vol 1, the author defines a "subobject of $A$" as "an equivalence class of monomorphisms with codomain $A$" (for a suitable notion of equivalence). He then defines what it means for a category to be well-powered: "$\mathcal{A}$ is well-powered when the subobjects of every object constitute a set". Thus, for instance, the category of sets is well-powered.
I'm having trouble understanding exactly what it means to have such a set of subobjects. As far as I can tell, each element of a set should also be a set, but an equivalence class of monomorphisms could be a proper class: for instance, the class of singleton sets is not a set. On the other hand, it seems that one can cheat by defining a subobject to be a class containing one representative of each equivalence class of monomorphisms, even though this is not, strictly speaking, what's stated in the book.
How can one solve this problem? Is there a "normal" set theory where such a set of subobjects can contain proper classes? Or does one need to cheat like suggested above?
No, proper classes are not elements of other sets (or other classes, for that matters).
But this is the great power of universes. Classes of one universes are just sets in a larger universe. So when you want to talk about collections of classes, you move to a larger universe, where you can treat them formally as sets.
Another, more complicated method of solving this issue, is to talk about schema of definitions when it comes to equivalence, and then the collections are represented by one of the classes (and you can sometimes prove that this representative doesn't matter). Then the whole thing becomes a much more technical and involved from a formal point of view, which is not a bad thing. But I think that if you're interested in category theory and want to talk about larger and larger categories, then universes are probably the way to go.