After delving a little bit into set theory, I noticed that most of its theorems revolved around whether $A$ is an element of $B$ ($A \in B$).
For example, if we gather enough facts about what elements are contained in the set of natural numbers ($\mathbb N$) and which sets containing $\mathbb N$, we could build Peano Arithmetics from such facts, which means, we could build the foundations of mathematics without proving existences of anything.
This is completely fine if those objects are truly exists. But the lack of truth about this matter makes me really uneasy. It's like we could swap those objects with something like God, or four-horned unicorn, and it won't matter at all.
So my question is, how do mathematicians justify this kind of mathematics?
Firstly, not all mathematicians are satisfied with set theory as it stands.
Research into type theory and in particular homotopy type theory has created an alternate foundation for mathematics which is highly attractive in many ways, including that it opens up computer verification of proofs as a realistic possibility for the future of mathematics.
Category theory has also been proposed as an alternate foundation for mathematics; indeed, much of mathematics can be carried out in categories known as "toposes", and Lawvere famously axiomatised the "category of sets" as a well-pointed topos satisfying the axiom of choice with a natural numbers object. In this axiomatisation, the primitive notions are of sets and functions between sets; "elements" are derived from these notions.
How one "interprets" any mathematical theory is more of a philosophical question than a mathematical one. There are three main schools of thought. Allow me to drastically oversimplify them as I explain them.
Finitism states that the only objects which "actually exist" are natural numbers. All the complicated machinery mathematicians imagine involving infinite sets is really only useful as a way to prove things about the natural numbers. There's often some overlap between this school and that of formalism; David Hilbert is a famous example of someone who falls within both.
Formalism posits that mathematics is just manipulation of symbols according to some set of rules. According to formalists, the fact that something can be proved under ZFC doesn't tell us anything more than that it can be proved under ZFC. By this view, it wouldn't matter in the slightest if we decided to call the set of all sets "God"; clearly, the fact ZFC tells us no such set exists says nothing about whether "God" as humans customarily think about the concept exists.
Platonism states that the mathematical entities studied in (for example) are "real" in some sense, and that every statement in ZFC is either "really true" or "really false" regardless of whether the statement can be proved or disproved. The fact that ZFC can't tell us whether the Continuum hypothesis holds simply means, according to this view, that ZFC is an incomplete axiomatisation of a very real phenomenon of "sets". The question of what "actually exists" that ZFC is trying to describe is a rather complex one. Judging by your comments, you seem to agree in principle with the Platonists that we should be trying to do mathematics that describes "real things"; you're just not sure sets are "real".