In Henning Makholm's answer to the question, When does the set enter set theory?, he states:
In axiomatic set theory, the axioms themselves are the definition of the notion of a set: A set is whatever behaves like the axioms say sets behave.
This assertion clashes with my (admittedly limited) understanding of how first-order logic, model theory, and axiomatic set theories work. From what I understand, the axioms of a set theory are properties we would like the objects we call "sets" to have, and then each possible model of the theory is a different definition of the notion of a set. But the axioms themselves do not constitute a definition of set, unless we can show that any model of the axioms is isomorphic (in some meaningful way) to a given model.
Am I misunderstanding something? Is the definition of a set specified by the axioms, or by a model of the axioms? I would appreciate any clarification/direction on this.
Update: In addition to all the answers below, I have written up my own answer (marked as community wiki) gathering the excerpts from other answers (to this question as well as some others) which I feel are most pertinent to the question I originally posed. Since it's currently buried at the bottom (and accepting it won't change its position), I'm linking to it here. Cheers!
The axioms of ZFC (or any other sufficiently strong first-order formal system) cannot define the notion of "set", in the sense that you're looking for, namely that ZFC cannot pin down a unique structure that satisfies ZFC. Why so? Because ZFC cannot prove its own consistency, by Godel's incompleteness theorem, and hence ZFC cannot prove that there is a model of ZFC. Furthermore, ZFC can prove that if ZFC is consistent then it has infinitely many models, not at all a single one.
Similarly, no recursive extension of first-order PA can (completely) define the natural numbers, because we can prove (in our meta-system that is usually ZFC) that PA has non-standard models. However, second-order PA is categorical (has a unique model up to isomorphism) and arguably captures completely the natural numbers. The catch is that you need to be in a meta-system that already has the standard model of PA before you can prove this fact about second-order PA, so in a way there is a priori no way to define the natural numbers.
You might want to read this post that I wrote about what every usable formal system (as of today) ultimately depends on, which cannot be further broken down into simpler notions.
There's a partial way to get around the circularity, that appears to be what many mathematicians do in practice. We can use natural language and define "set" to be a type of object such that the ZFC axioms hold, and insist that we can only call something a set when we have proven its existence in ZFC. Note that there is no need to have a model of ZFC here, because we're saying that if you can't prove it then I don't accept that it exists (but neither am I insisting that it doesn't), so it becomes a purely syntactic notion.
In other words, if we define "set" syntactically using the axioms of ZFC, then we've escaped most of the circularity (except what we already need to know about string manipulation). What we still can't escape is that we can't define "model" in the usual sense without collections of some sort, and so we can't even articulate that ZFC has successfully defined any structure. (Unless of course our natural language is so powerful, but then we're in trouble.)