I was reading a proof of the following theorem in my textbook:
A set $A$ is closed iff $A' \subseteq A$.
Proof: Suppose $A$ is closed and $x \in A'$. If $x \notin A$, then $x\in A^c$, an open set. Thus $\mathcal{N}(x, \delta)\subseteq A^c$ for some positive $\delta$. But then $\mathcal{N}(x, \delta)$ can contain no points of $A$. Thus $x$ is not an accumulation point of $A$ and so $x\notin A'$, a contradiction. We conclude that $x\in A$. Therefore $A'\subseteq A$.
Now suppose $A'\subseteq A$. To show that $A$ is closed, we show $A^c$ is open. If $A^c$ is not open, there is $x\in A^c$ that is not an interior point of $A^c$. Therefore, no $\delta$-neighborhood of $x$ is a subset of $A^c$; that is, each $\delta$-neighborhood of $x$ contains a point of $A$. This point must be different from $x$, since $x\in A^c$. Thus $x\in A'$. But $A'\subseteq A$, so $x\in A$. This is a contradiction. We conclude that $A$ is closed.
Somewhere along the proof, the complement of $A$ is proven to be open. In order to achieve this, the author first assumes that it's not open and produces a contradiction. But isn't that wrong since it's possible for a set to be neither open nor closed?
No problems here. Note that the author doesn't assume $A^c$ to be closed. Just that it isn't open. And since "open" means "for any point in the set, we have a certain property", we get that "not open" means "for at least one point in the set, we don't have that property". The point $x$ they choose is one such point guaranteed to exist by that assumption.