Consider a curve $\omega$ in a metric space $(X,d)$, i.e (a continuous w.r.t $d$) function $\omega:[0,1] \to X$.
Why, unless $X$ is a vector space, does its velocity $\omega'(t)$ not have a meaning/cannot be defined ?
Is it just because the usual way we would define $\omega'(t)$ is $$\lim_{h\to 0} \frac{\omega(t)-\omega(t+h)}{h},$$
and we wouldn't have a notion of $\omega(t)-\omega(t+h)$? Or is there a deeper reason I am not understanding?
Succinctly, the answer to your question depends strongly on what type of a structure all velocities you would want/require to assemble to.
Based on your other question Is my proof that a constant speed geodesic is a geodesic correct? I am assuming your question is based on Box.5.1 of Santambrogio's book Optimal Transport for Applied Mathematicians (pp.187-188). There "velocity not having any meaning for metric spaces" is used to motivate the general program of extracting geometric information from Lipschitz property (roughly speaking). Note that the desired geometric information is to have some (a priori) linear structure in this case.
Just as one could say "out of a definition of velocities using linear algebraic operations there emerges a linear space structure" one could also say "if one wants a linear space structure for velocities one ought to use linear algebraic operations to define velocities". One can then add, "if velocities are to be local, then it should be sufficient to have linear algebraic operations that are defined locally", which now includes differentiable manifolds.
An alternative formalism that is intrinsic to metric spaces is the so-called tangent cone construction. Without going into details, this is a construction analogous to the tangent space construction, where only the distance and homotheties are used (and Gromov-Hausdorff convergence), and the end result is a metric space ($\ast$). See e.g. the discussion at https://mathoverflow.net/a/363777/66883. There is some fine print involved here (a great starting point for all this is Burago, Burago & Ivanov's book A Course in Metric Geometry), but just to give some justification, an exercise from the aforementioned book states that the tangent cone at any point $p$ of a Riemannian manifold $M$ of dimension $d$ exists and is isometric to $T_pM\cong \mathbb{R}^d$ (Exr.8.2.4 on p.275). (There is also a related notion of "space of directions".)
($\ast$) A priori, that is (which phrase I've used vaguely throughout). Under certain conditions there is extra algebraic structure; e.g. if $M$ is a sub-Riemannian manifold then the tangent cone exists at any point in an open and dense subset and has a natural nilpotent Lie group structure (by a theorem by Mitchell; see his paper "On Carnot-Carathéodory metrics").