I'm doing exercise 4.1(e):
The answer key says that the relation above is an equivalence relation. I don't even know what the relation they've defined means.
This is how I'm interpreting: Let $ \in$ define the relation. I believe "in" means $\in$.
Pf: Reflexive $\iff$ Symmetric
$\emptyset$ $ \in$ $\emptyset$
Transitive: if $\emptyset$ $ \in$ $\emptyset$ and $\emptyset$ $ \in$ $\emptyset$, then $\emptyset$ $\in$ $\emptyset$ .
therefore $ \in$ is an equivalence relation?
Endpf.
Explanation requested.
Someone of doubt also claims that
" $ \emptyset$ in A, where A is a non-empty set"
is an equivalence relation?. I have no idea what "in" means.

This is indeed somewhat confusing; let me break down what they're talking about.
The source of your confusion is I think the use of the word "in." In this context, it does not mean "$\in$." When Hrbacek/Jech say "Is $E$ a [someting] relation in $X$?," it might be better for them to use the word "on" - what they mean is that $X$ is the underlying set, and $E$ is the relation. I'll be using their meaning of the word "relation in $X$" throughout this answer, even though I disapprove.
Now what about the specific problem(s) here? Well, these exercises are designed to do two things. The first is to show you that the choice of the underlying set can matter: e.g. the relation $E=\{(a, a)\}$ is reflexive in $\{a\}$ (since every element of $\{a\}$ is related to itself) but not in $\{a, b\}$ (since $b$ is not related to itself: $(b, b)\not\in E$).
The second point is to give you some practice reasoning in a nonintuitive context - specifically, about the empty set. $\emptyset$, as usual, is a confusing object. Remember that "for all" statements about the empty set are vacuously true - for instance, $\emptyset$ is a set of elephants, since every element of $\emptyset$ is an elephant. More relevantly here, $\emptyset$ is a set of ordered pairs, hence a relation (it's the relation which never holds).
Now as long as our domain isn't stupid, $\emptyset$ clearly won't be reflexive: no element will be related to itself, since no element will be related to anything (our relation never holds). But what if we take our underlying set to also be $\emptyset$? Then every element of $\emptyset$ is related to itself - remember that "for all" statements about $\emptyset$ are vacuously true!
So $\emptyset$ is a reflexive relation if our domain is $\emptyset$. And it's clearly transitive and symmetric no matter what the domain is (exercise). So, to use the phrasing of the book, $\emptyset$ is an equivalence relation in $\emptyset$.
Re: the end of your question, you say that
This is false - $\emptyset$ isn't reflexive in any nonempty set $A$ (why?). Where does the book claim this? ${}{}$