The differential $\mathrm Df(x)$ of a map $f$ at a point $x$ between normed vector spaces $V$, $W$ can be regarded as the linearization of this map in an neighborhood of $x$, that is
$f(x+h) = f(x) + \mathrm Df(x) [h] + o(||h||)$ (1)
This is clear, but now for manifolds, the differential acts on the tangent space per
$\mathrm Df(p):T_p M\to T_{f(p)}N, v\mapsto \mathrm Df(p)[v]$
where $p\in M$ and $f:M\to N$ , each manifolds
So it makes no sense to write something like $f(p+v)$ since $p\in M$, but $v\in T_p M$. I know how to define the differential using charts in local coordinates by $y\circ f\circ x^{-1}$ which is a map between $\mathbb R^d$. And we can apply everything whats is known from above.
But what is the equivalent of (1) for manifolds?
I thought of something like: for $v\in T_pM$, the point $\exp(v)$ is something like a linearized result?
Let us think of a chart as identifying an open neighbourhood of $0$ in $T_p M$ with an open neighbourhood of $p$ in $M$ (sending $0$ to $p$ and being ``identity" on the tangent spaces at those points), and doing the same for $N$ your map $f$ gives a map between open neighbourhoods of $0$ in the vector spaces $T_p M$ and $T_{f(p)} N$, and then the differential will be the differential of this map in the previous sense, as you said. One then checks that this does not depend on the choice of chart in the above sense.
But more philosophically speaking, I guess you are right in that the linearization of manifolds (such as $M$) is much more subtle than the linearization of maps (such as $f$). This should be a-priori clear, as a manifold is an object in a category, while a map is an element in a set, so for example when we try to imagine the ``variation" that we are trying to linearize, in the manifold situation we immediately start thinking about the tangent bundle, i.e. varying the tangent space - varying vector spaces, while in the original case vector spaces are fixed, and we just vary stuff inside of them. Of course this is hidden in the formalization via charts, and one can write everything, check everything, without thinking too much, but I guess it is probably worthwhile to contemplate, like you seem to do. For example, probably when one arrives to connections and all that is related, the need for such a contemplation in order to get a good feeling of things becomes much more transparent.