I am reading through Tao's Analysis.
Tao defines all these rules for sets and they can be expressed in terms of first order logic, hence ZFC being a "first order theory." But then these "underlying" notions of sets, this idea that a set is just a kind of special "thing" that can "contain" other "things" (including other sets), is this technically part of ZFC or something more fundamental? Is "set theory" considered a foundational framework that by itself doesn't depend on anything deeper like propositional or first order logic?
Set theory is the theory of sets. The essential properties of sets are as follows:
There's a relation $a\in b$ where both $a$ and $b$ can be sets. In the most commonly used set theory, Zermelo-Fraenkel set theory with Axiom of Choice, or short ZFC, sets are the only things that can be in those relations, but other set theories allow additional things that can only be put on the left side, called urelements, and/or things that can only be put on the right side, called proper classes. Note that the term “proper class” is also used in the context of ZFC, but there they are not actually objects of the theory.
When $a\in b$, we say $a$ is an element of $b$.
A set is completely determined by its elements. That is, if there are two sets $a\ne b$, then there exists an element $c$ such that exactly one of $c\in a$ or $c\in b$ is true. Usually that's stated explicitly as axiom, called axiom of extensionality.
There are ways to construct sets from a large class of logical formulas, such that something is an element of that set iff the formula is true. Naive set theory just allowed any formula, but that leads to contradictions. Different set theories have different restrictions on which formulas lead to sets. In ZFC, the restriction ultimately boils down on allowing only formulas of the form $\phi(x,y)\land y\in a$ where $a$ is a set, and $\phi$ is a formula that for each $y$ is true only for one $x$.
Note that a set theory can also provide other, more explicit ways to construct sets (for example, ZFC contains, besides others, the axiom of the power set, that states that for any set also its power set exists), or outright state the existence of sets with specific properties (in ZFC, the axiom of infinity effectively states the existence of an infinite set). However there are set theories where the general rule is sufficient, like New Foundations.
Whenever you've got a theory that fulfils those points, it would be called a set theory. But usually, if you say set theory without any qualifications, you mean ZFC.
Now as the last point already shows, set theory very much depends on having a logical system (which usually, as in the case of ZFC, will be first-order logic). But then, I have a hard time thinking of any mathematical theory that doesn't.