I am currently studying a course on ZF Set Theory and would like to know how to show that a set does not exist using the axioms.
For example, the following two sets are clearly equivalent to a set of all sets (and so cannot exist), however, I am unsure how to prove this rigorously using the axioms to show that such sets cannot exist:
$$\{x : \exists y (x \in y) \} \space \text{and} \space \{x : \exists y (x = y) \}$$
My intuition says that this is a result of The Axiom of Separation. This states that for a WFF (well formed formula) $\phi$, we have the following relation:
$$ \forall a_1 ...\forall a_n \forall z \exists y\forall x ((x \in y) \leftrightarrow (x \in z) \land \phi(a_1...a_n))$$
In other words, this states that every property specifies subsets of a particular set. In the case of the two examples above $\phi$ would be $\exists y (x \in y)$ or $\exists y (x = y)$ respectively.
Clearly, some bounding set is needed in the two examples that I provide above in order for these to be valid sets, although I am struggling to make this argument rigorous and would be grateful for any guidance in producing a stronger argument to show that these sets cannot exist under the axioms of ZF Set Theory.
There isn't a general algorithm for proving something is not a set. One approach is using a proof by contradiction. Assume it is a set, and derive a contradiction.
Remark: The Class $V = \{x \space| \space x = x\}$ is not a set. Note: We would call V a Proper Class.
I will prove the following claim: "$\{x \space |\space \exists y(x∈y)\}$ is not a set."
Proof by Contradiction:
Assume $A = \{x\space |\space ∃y(x∈y) \}$ is a set (so that we can elicit a contradiction). By Corollary we can find some set $x_0$∉A.
By the Pairing Axiom, since $x_0$ is a set we have that:
$$y = \{x_0,x_0\} = \{x_0\} \space \text{is a set} $$
So, $x_0 \in y$. Thus, the existence of $y$ means that $x_0 \in A$. This is a contradiction as we picked $x_0$ so that it is not in $A$.
Note$_1$: The proof for showing that $\{x \space | \space ∃y(x=y)\}$ is not a set is very similar (Hint: instead of using the Axiom of Pairing, use the Axiom of Extentionality).
Note$_2$: Your intuition about the Axiom of Separation is correct, you could use it directly. The proof would be a more complicated version of using Separation to prove $V$ is a not set.