I am wondering whether the category of sets, $\mathbf{Set}$, and the category of groups, $\mathbf{Grp}$, are well defined or not.
Suppose we choose a singleton $\{a\}$ from $\mathbf{Set}$. And suppose we choose a singleton $\{b\}$ from $\mathbf{Set}$. How could we possibly know if we have chosen the same singleton or not?
Similarly, if we refer to a singleton $\{a\}$, then how could we possibly refer to it a second time and be convinced that it is indeed the same singleton?
How can I know if $\{\text{“horse”}\}$ and $\{\text{“caballo”}\}$ are different singletons, or simply two expressions of the same singleton?
The same questions come up with trivial groups. Suppose I choose two trivial groups. How do I know if I have chosen the same one twice or not?
And how can I tell whether or not the category $\mathbf{Set}$ contains multiple instances of the same singleton?
It seems that there is no possible language that could make sense of the objects of $\mathbf{Set}$. In that case, it seems fair to admit that $\mathbf{Set}$ is, on some level, ill defined. Please correct me! or please affirm me.
A similar question was asked earlier, How to treat isomorphic objects in a category?, but I don't think the answers address my concerns.
Personally, it would make sense to me if the objects in $\mathbf{Set}$ that are equivalent were defined to be equal, as sets. But that's apparently not the way it works in category theory. You can have many distinct sets that are all equivalent, from what I understand.
One of the strengths of category theory is that indeed it gives a convenient framework to define all sorts of notions of "equivalence", some stronger than others.
For instance, you have the notion of isomorphism between objects of the same category: in the category of sets, being isomorphic means having the same cardinality; in the category of groups, it is having the same structure as groups. So all trivial groups are indeed "equivalent" in this sense.
On the other hand, there is one notion that does not change meaning within category theory: equality. Equality is hardwired into mathematics (at least in the framework commonly used by contemporary mathematicians): two objects are equal if they are, very literally, the same. It is a primitive notion. So two singletons $\{a\}$ and $\{b\}$ are always isomorphic in the category of sets, but they are equal only when they are the very same set, meaning that they have the same elements (this is what characterizes a set), so in this case meaning that $a=b$.
Your example with words is sort of non-mathematical. If you define words as sequences of symbols, then "horse" is not the same word as "caballo", so the sets $\{\text{"horse"}\}$ and $\{\text{"caballo"}\}$ are different. If you define a word as a sort of semantic equivalence class, well... give me a correct mathematical definition of what it means and maybe I can say whether "caballo" and "horse" are the same. But the problem here is not with the category of sets, it is that you usage of words as objects is not very well defined.