This is really a question of definitions. If a function $f$ is not defined on an open set containing $x$, how do we define the derivative of $f$? Is it sufficient to be locally approximable by linear functions on its points of definition, or do we consider the function not differentiable here? I am interested in defining a function $D:\Bbb R^{\subseteq\Bbb R}\to\Bbb R^{\subseteq\Bbb R}$ such that $x\in\operatorname{dom}D(f)$ and $D(f)(x)=f'(x)$ whenever $f$ is differentiable at $x$. When $f$ has a peculiar domain in the neighborhood of $x$, the exact definition of the limit is confusing me. What is the derivative at an isolated point?
Edit: Here is my proposed definition for a derivative, defined as a predicate on possible limits, that makes sense when $x_0$ is an accumulation point of $A$, the domain of $f$, even if $x_0\notin A$:
$f$ has derivative $L$ at $x_0$ iff for all $\varepsilon>0$, there is a $\delta$ such that for any $x,y$, $$x,y\in A\cap B(x_0,\delta)\wedge x\ne y\implies\left|\frac{f(x)-f(y)}{x-y}-L\right|<\varepsilon.$$
This definition actually makes sense even if $x_0$ is an isolated point or an exterior point of $A$, but in these cases $f$ has every number as a derivative (by contrast to when $f$ is genuinely not differentiable in the conventional sense, in which case no number is a derivative). If one adds the proviso "$x_0$ is an interior point of $A\cup\{x_0\}$", one recovers the conventional definition, except that the point $x_0$ is not required to be defined, which allows you to patch over removable singularities.
The standard definition of ''$f$ differentiable at $x_0$" usually includes a requirement that $x_0$ is an interior point of the domain of $f$, but really the limit that defines the derivative makes sense as long as $x_0$ is a point in the domain of $f$ that is also an accumulation point of the domain of $f$.
One issue that could arise is that a function that is not differentiable in the usual sense could have two different derivatives when restricted to two different subdomains. For instance, in your example you define $E = \displaystyle \left\{ \pm \frac 1n : n \in \mathbb N \right\}$. Suppose that $f(x)$ is defined as $f(x) = x$ if $x \in E$ and $f(x) = 0$ otherwise. Then $f|_{E \cup \{0\}}(x) = x$, and $f|_{\mathbb R \setminus E}(x) = 0$. In this case $$\left(f|_{E \cup \{0\}} \right)'(0) = 1,$$ but $$\left( f|_{\mathbb R \setminus E} \right)'(0) = 0.$$