In my lecture notes the total differential is defined as follows:
Definition 1: Let $U \subset \mathbb{R}^{n}$ be open. A mapping $f : U \to \mathbb{R}^{m}$ is differentiable at $x \in U$ if there is a linear mapping $L_{x}:\mathbb{R}^{n} \to \mathbb{R}^{m}$ such that for all $h$ with $x + h \in U$,
$f(x + h) = f(x) + L_{x}(h) + r_{x}(h)$ with $\lim \limits_{h \to 0} \frac{r_{x}(h)}{\|h\|} = 0$
In this case we call $L_{x}$ the differential of $f$ at $x$.
In this definition we require $U$ to be open because we need to be able to approach $x$ since otherwise any linear map would satisfy this definition and the differential would not be unique.
For example, consider the map $F : M \to \mathbb{R}, F=0$ where $M=\{(x,y) : x \in \mathbb{R}, y=0\}$, then we can only consider $h$ of the form $(h_{1},0)^T$ since otherwise we would leave the domain $M$ of the function. Now let $A_{\lambda}=(0,\lambda), \lambda \in \mathbb{R}$ be a $2 \times 1$ matrix, then any linear map $L_{\lambda}=A_{\lambda}h$ satisfies the above definition, so the differential is again not unique. It is easy to see that $M$ is not open. Consider an open ball $B_{r}(x,y)$ around an arbitrary point in $M$. Then this ball is not contained in $M$ for any $r$ because there always exist points with $y \neq 0$ in $B_{r}(x,y)$.
Intuitively, the reason why the differential is not unique is that we cannot approach $(x,y)$ from abitrary directions. In fact, in the example we can only approach $(x,y)$ in the direction of the $x$-axis.
Now the next idea one might have is to alternatively require $x$ to be a limit point (which works fine in the one-dimensional case). But this also does not work since it still might be the case that we cannot approach $(x,y)$ from arbitrary directions (and in fact the same example as before works to show this since any point in $M$ is a limit point of $M$).
I've also found a proof that the differential is unique if the function is defined on an open set. The proof goes like this.
Proof: Let $L,M$ satisfy Definition 1.
First note that $\lim \limits_{h \to 0} \frac{\|r_{x}(h)\|}{\|h\|}=\|\lim \limits_{h \to 0} \frac{r_{x}(h)}{\|h\|}\|=0$ since the norm is continuous and only $0$ for the zero vector.
Then $\lim \limits_{h \to 0} \frac{\|L(h)-M(h)\|}{\|h\|}=\lim \limits_{h \to 0} \frac{\|L(h)+f(x)-f(x+h)+f(x+h)-f(x)-M(h)\|}{\|h\|}$
$\leq \lim \limits_{h \to 0} \frac{\|L(h)+f(x)-f(x+h)\|}{\|h\|} + \lim \limits_{h \to 0} \frac{\|f(x+h)-f(x)-M(h)\|}{\|h\|}=0$
Now let $v \in \mathbb{R}^{n}:h=tv, t \in \mathbb{R}$. Note that $L(tv)=tL(v)$ and $M(tv)=tM(v)$ by linearity. Since $tv \to 0$ as $t \to 0$ It follows that
$0=\lim \limits_{t \to 0} \frac{\|L(tv)-M(tv)\|}{\|tv\|}=\lim \limits_{t \to 0} \frac{|t| \|L(v)-M(v)\|}{|t| \|v\|}=\frac{\|L(v)-M(v)\|}{\|v\|}$
Again since the norm is only $0$ for the zero vector, this proves that $L(v)=M(v)$, so $L$ and $M$ are in the fact equal.
Now to my question. I cannot seem to understand where this proof breaks down if we instead assume that $x$ is a limit point of the domain of the function $f$. Obviously it cannot work in general in this case as can be seen from the example above (and there are many more). Is it because we assume $h:x+h \in U$, so we cannot show that $L(h)=M(h)$ for $h$ that do not satisfy this property?
From the start of the problem we ask that $x + h \in U.$ In particular this means that when we substitute in $h = tv$ and deduce
$$ \Vert M(v)-L(v)\Vert = 0 \iff M(v)=L(v) $$ then this is only valid for $v$ such that the substitution itself is valid. That is we must have $x + tv \in U$ while $t \to 0$. In the case where $U$ is open we can do this for any $v$ ( just choose $t$ small enough !) hence we get
$$ M(v) = L(v) \quad \forall v\in \mathbb R^n.$$ That is, we find $M = L$.
If $U$ is not open then issues can arise. For example for $U = \mathbb R \times \{0\}$ then as you rightly pointed out we have the constraint that $h$ is always of the form $(h_1,0)^T$ so instead of getting $ \Vert M(v)-L(v) \Vert= 0$ for all $v \in \mathbb R^2$ we get
$$ M(v) = L(v) \quad \forall v = (v_1,0)^T$$ which is not enough to deduce that $M = L$.
However it is still possible to get unicity in some cases (even if $U$ is not open) I haven't checked the details but a closed disk $U =\{x \in \mathbb R^m : \Vert x \Vert \leq 1 \}$ should be such an example.