You may refer to this question for a formal statement of Rudin's Theorem 2.34 and the proof he provides.
Let $E \subset X$, where $X$ is a metric space and $E$ is "non-closed", i.e. it could be either "open" or "neither open nor closed". Since, $E$ is non-closed there is atleast one point $p\in X$ such that $p \notin E$ and $p$ is a limit-point of $E$. With that said, consider the following intuition
Intuition: Let $N_r(p)$ be an open-ball centered at $p$ and of radius-$r$. Then, portion of every neighborhood of $p$ that falls inside $E$ can be constructed as a union of infinitely many open sets, i.e. $N_r(p)\cap E = \cup_{\alpha}\,G_{\alpha}\,\,\forall r>0$. Then from this, we can always construct an [infinite] open-cover of $E$, as $\{\,G_{\alpha}\}\cup\{E\,\backslash\,(\cup_{\alpha}\,G_{\alpha})\}$; whose union is equal to $E$. And any finite subset of this open-cover will leave-out portions of $N_r(p)\cap E$, and hence won't be an open-cover of $E$.
Is this intuition right? And is there a better way to formalize this?