I know that $\sqrt{2}\not\in\mathbb Q$ and $\sqrt{2}\in\mathbb R$ but it is not obvious to me why $\{p\in\mathbb Q : 0<p<\sqrt{2}\} \subset \mathbb R$ is not open. If it is not open, it means its complement is open i.e.
$$A:=\{p\not\in\mathbb Q : 0<p<\sqrt{2}\}\subset \mathbb R$$
where each rational number is surrounded by two real numbers and where $p\not\in\mathbb Q$ basically means $p\in\mathbb R/\ \mathbb Q$ (i.e. the rational section out). Now the culminating points are all in $A$ i.e. making it closed and it is also open because you can cover it with open balls (n.b. rational-real-rational -cover). So $A$ is clopen. Now what is $A^{C}$ then?
Is there some term for not-clopen?
Let $S=\{p\in\Bbb Q:0<p<\sqrt2\}$. Not only is $S$ not open, it has empty interior: it does not contain any non-empty open set. Suppose that $U$ is a non-empty open subset of $S$; the open intervals form a base for the topology of $\Bbb R$, so $U$ must contain a non-empty open interval $(a,b)$. But then $a$ and $b$ are real numbers such that $(a,b)\subseteq S\subseteq\Bbb Q$, which is impossible: there is certainly an irrational number between $a$ and $b$.
For essentially the same reason your set $A$ is not open. If $(a,b)$ were a non-empty open interval contained in $A$, every real number between $a$ and $b$ would be irrational, which is of course false.
And since neither of these complementary sets is open, neither can be closed: if $S$ were closed, $A$ would be open, and if $A$ were closed, $S$ would be open. Thus, both $S$ and $A$ fail as badly as possible to be clopen.
I know of no single term that means not clopen; one simply says that $S$ is not clopen, unless one can specify more precisely that $S$ is not open, that $S$ is not closed, or (as in this case) that $S$ is neither open nor closed.
Finally, note that as André already mentioned, a non-open set does not necessarily have an open complement. A set with an open complement is closed, and there are many sets that are neither open nor closed. A few people have studied the class of door spaces, spaces in which every set is either open or closed (or both), but this is a very restricted class (and not really a very interesting one). Most spaces of interest are not door spaces.