Consider a differentiable finite dimensional manifold $M$ with $\text{dim}M = n$, modelled on $\mathbb{R}^n$. Given a metric tensor $g$, at any point $p \in M$, the tangent space $T_pM$ can be equipped with a norm $||v|| = \sqrt{g_p(v,v)}$. Thus, a curve $\gamma:\mathbb{R} \to T_pM$ can be differentiated (assuming the derivative exists) :
$$ \lim_{|h|\to 0} \frac{||\gamma(x+h)-\gamma(x)- Ah ||_{T_pM}}{|h|} = 0, $$ where $A: \mathbb{R} \to T_pM$.
Why I have never seen something like this ? Instead, the everyone uses the definition
$$ \gamma'(x) = \lim_{h\to 0} \frac{\gamma(x+h)-\gamma(x)}{h}, $$ independently of the existence of a metric. Why ?
The second definition is not independent of a distance function, as a limit in a metric space is defined using a distance function (in this case a norm), that is, the expression $\lim_{y\to x}f(y)=L$ for some function $f:A\to B$ is equivalent to
$$ \forall \epsilon >0,\,\exists \delta >0:0<d_A(x,y)<\delta\implies d_B(f(y),L)<\epsilon $$
where $d_A$ and $d_B$ are the distances in $A$ and $B$ respectively. In your case the expression $\lim_{h\to 0}\frac{\gamma (x+h)-\gamma (x)}{h}=\gamma '(x)$ is equivalent to
$$ \forall \epsilon >0,\, \exists \delta >0:0<|h|<\delta \implies \left\| \frac{\gamma (x+h)-\gamma (x)}{h}-\gamma '(x)\right\|_{T_pM}<\epsilon $$
what is exactly the same definition of the Fréchet derivative when $h$ is real.