Differential of vector field

Question

Given smooth manifolds $\mathcal M$ and $\mathcal N$ and a smooth map $f : \mathcal M \to \mathcal N$, we define the differential of $f$ at $p \in \mathcal M$ as the linear map \begin{align} \text{d}f_{p} : \mathcal T_{p}\mathcal M &\to \mathcal T_{f(p)}\mathcal N \\ \gamma'(0) &\mapsto (f \circ \gamma)'(0), \end{align} where $\gamma$ is a smooth curve on $\mathcal M$ satisfying $\gamma(0) = p$. Since a vector field is a smooth map $X : \mathcal M \to \mathcal T \mathcal M$ and $\mathcal T \mathcal M$ is itself a smooth manifold, why can't we just apply the above definition to $X$? Why do we instead need the notions of connections and covariant derivatives? Isn't $(X \circ \gamma)'(0)$ a perfectly sensible tangent vector in the tangent space of $\mathcal T \mathcal M$ at $X(p)$?