So many things get called "set theory" that I'm no longer sure the term is meaningful.
This isn't helped by the rather vague notion of "formalizable in ZFC" used to avoid detailed discussion of foundations in other areas of mathematics. To be sure, any recursive system can be implemented within ZFC (up to - I don't even know if there's a word for it - realizability? physicality?) using some encoding; but if we take this as license to say that everything is a part of "set theory", then we might as well say that "set theory" is actually arithmetic (which is essentially just Fortran, which is really a Magic: The Gathering format!)
Clearly, "set theory" means something more specific than "that which might serve as foundations," but even ignoring the vagueness with which things are/are not "really" a part of ZFC, the list of alternative set theories is long and varied, and I'm not sure that I can differentiate a "set theory"-like theory from a "set theory." For instance, I can't think of any obvious way to justify the claim "type theory is not set theory" that doesn't also suggest "New Foundations is not set theory."
So what makes a set theory a set theory? What are the common elements which are shared by all and only set theories?
Since set theory is principally concerned with the membership relation, another way to ask this question is "what properties uniquely characterize the membership relation '$\in$'?"or, if we prefer to leave '$\in$' ambiguous, "what properties must a membership relation have?"
I think a good starting point is that all set theories should agree on the behaviour of finite sets, since this is something which might be considered empirical fact. In other words, every set theory should at least be able to produce the schemas:
"There is an empty set" [$\neg(\zeta\in \emptyset)$]
"The union of two [finite] sets exists"; from which follows "the union of a [non-empty] finite set [of finite sets] exists" [$\exists x(\zeta\in x\iff(\zeta\in\xi\lor\zeta\in\gamma))$]
"The intersection of two [finite] sets exists"; from which follows "the intersection of a [non-empty] finite set [of finite sets] exists" [$\exists x(\zeta\in x\iff(\zeta\in\xi\land\zeta\in\gamma))$]
"The relative complement of two [finite] sets exists" [$\exists x(\zeta\in x\iff(\zeta\in\xi\land\neg(\zeta\in\gamma)))$]
"The set containing a finite set as it's sole member exists" [$\exists x(\zeta\in x\iff \zeta=\xi)$]
Beyond this extremely weak requirement, I'm not sure what else a set theory needs.
Comment: the explicit declaration of the constant '$\emptyset$' was deliberate, and necessary to avoid universal quantifiers. This was done for philosophical reasons, in keeping with the theme of set theories agreeing on "empirical facts." If I give you a bag, you cannot be certain that it is truly empty, only that the bag does not contain any particular thing which you might name. It is entirely possible that the bag contains something which you are simply unaware of - a fine powder, a gas, massless dark matter, an actual ghost - but you cannot say whether or not these things are in the bage because you are unaware of them. This also means that we could have a theory with more than one empty set, but then whether or not that still counts as a "set theory" would require an answer to this question.
This is really a "you know it when you see it" sort of situation. Some theories are obviously set theories, some are obviously not, and others lie somewhere in between, so that some people would describe them as set theories and others would not. I don't think it's necessary (or possible) to have a precise definition of what counts as a set theory.
... but if pressed, I would propose the following definition. Here I will focus on what are called "material set theories". As Eric Wofsey points out in the comments, there are also "structural set theories", like ETCS, which arguably also deserve the name "set theory". Anyway, here's my "definition":
A material set theory is a theory in a language containing a binary relation $\in$ (membership), with a specified formula $S(x)$ (intuitively meaning "$x$ is a set"), which proves the following sentence: $\forall x\forall y (S(x)\land S(y)\land (\forall z(z\in x\leftrightarrow z\in y))\rightarrow x=y)$ (extensionality).
The point is that the core of the concept of "set", for me, is already captured by saying that a set is determined uniquely by its elements. Beyond this, different set theories could disagree on questions of which sets exist.
In a set theory like ZFC, where everything is a set, the formula $S(x)$ can be a validity ($\top$ or $x=x$). But other set theories may have things that aren't sets (e.g. classes or atoms). You might argue that a theory that has too much other stuff built in is really a "set and other stuff theory". But I'm erring on the side of giving a maximally inclusive definition.