The axiom in infinity in ZF states the following:
$$\exists x ((\emptyset \in x )\land \forall y (y \in x \implies y\cup \{y\} \in x) $$
My notes then defines
$$\mathbb{N}:= \bigcap\{z \in \mathcal{P}(x): \emptyset \in z \land \forall y(y \in z \implies y \cup \{y\} \in z)\}$$
But this seems to depend on the set $x$, so is this actually well-defined? I.e. if I choose another set $x'$ satisfying the same property and I define $\mathbb{N}$ w.r.t. $x'$, then will I even get the same set?
No, this does not depend on the set $x$.
If $x$ and $x'$ both have this property, then $x\cap x'$ also has this property. It follows that $\Bbb N$ is defined by intersecting over a subset of both $x$ and $x'$, which means that it does not depend on the choice of $x$.