Update: There are no counterexamples. This question stemmed from my confusion over the definition of "open relative to." Sorry if it is muddled. If you are similarly confused over the meaning, you might find it enlightening to read my response below as well as my proof of the second part of Theorem 2.30 posted here: Question about the proof of Rudin's Theorem 2.30
What would be a concrete example where $E$ is not open relative to $Y$?
A few definitions from Rudin:
Suppose $E \subset Y \subset X$ where $X$ is a metric space. To say that $E$ is an open subset of $X$ means that to each point $p \in E$ there is associated a positive number $r$ such that the conditions $d(p,q) < r, q \in X$ imply that $q \in E$.
Additionally, $E$ is open relative to $Y$ if to each $p \in E$ there is associated an $r > 0$ such that $q \in E$ whenever $d(p,q) < r$ and $q \in Y$.
In the question in your title, you start by saying
This is slightly ambiguous since it is not clear whether you mean that $E$ is given to be an open subset of $X$ or an open subset of $Y$. I assume you mean the former, so it is better to say
Now, since $E$ is an open subset of $X$, for every $p \in E$ there exists $r > 0$ such that $d(p,q) < r$ for $q \in X$ implies that $q \in E$. So, the same value of $r$ works to show that $E$ is relatively open in $Y$, because if $d(p,q) < r$ for $q \in Y \subset X$, then $q \in E$ by the previous statement, and $E \subset Y$ so $q \in Y$. Thus, there is no example of the kind that you seek.
This does not make Theorem 2.30 superfluous, however.
Some open subsets of $Y$ could indeed be nothing but open subsets of $X$ that happen to be contained in $Y$. I assume this is where your intuition is taking you when you said in the comments
But the fact is there are other subsets of $Y$ that are not of this kind. For instance, let $X = \Bbb{R}$, $Y = [0,1]$ and $E = [0,1)$. Clearly $E$ is not an open set in $X$, but it is open relative to $Y$ because $E = Y \cap (-1,1)$. So, the answer to your question
is yes, $E$ is not assumed to be an open set in $\mathbf{X}$ in the hypotheses of the theorem. Let me again emphasise that you must specify which is the ambient space when you say something is or isn't an open set.