The following definition of the derivative of a function is taken from Analysis on Manifolds by James Munkres:
Now my question is the following: Is the "neighborhood of $a$" an open set $U⊆A$ in $\mathbb{R}$ (the parent space) containing $a$ or is $U$ open in the subspace $A$?
I'm assuming that $U$ must be open in $\mathbb{R}$ as opposed to the subspace $A$ because then we could have all kinds of wacky domains (e.g a discrete subset of $[0,1]$ for the the function $f$ in which case the derivative wouldn't even exist (because the limit wouldn't even exist)).
In that case we can only take the derivative of functions that are defined on "nice-ish" sets (i.e. sets that have non-empty interiors). Am I correct in this observation?
Yes, when it says "a neighborhood of $a$" it means a neighborhood in $\mathbb{R}$. By this definition, the derivative of a function at a point is only defined when the point is in the interior of the domain of the function.
(This requirement is not at all intrinsic to the notion of "derivative", and you could define derivatives without requiring $f$ to be defined on an entire neighborhood of $a$. But it is convenient to have an entire neighborhood as part of the definition so that you don't have to constantly state extra hypotheses for your theorems, many of which involve such a neighborhood.)