A stalk is a colimit of all $\mathcal{F}(U)$ over all open sets $U$ containing $p$.
A stalk is alternatively thought of as the collection of germs at a point $p$. Hence, it cannot be thought of as the collection of all differentiable functions on a particular open set containing $p$ (it can be thought of as the collection of differentiable functions on the intersection of all open sets containing $p$, which may not be open). Hence, isn't it a contradiction that the colimit is not an object in the category?
Let us go back to the definition of the (pre)sheaf $\mathcal F$. It is defined as a functor $\mathcal F: \mathcal O \to \mathsf{NiceCat}$ where $\mathsf{NiceCat}$ is some nice, predefined category (you can look it up in your source).
At the heart of your question lies the premise that $\mathsf{NiceCat}$ is "the category of sets of differentiable functions on open sets" with some suitable kind of morphisms. Let us say for example that we want to regard these sets as rings, so that we're working in a subcategory of $\mathsf{Ring}$.
But this category is definitely not nice. We have to be very creative or work hard to determine whether some ring $R$ is a member of $\mathsf{NiceCat}$. This problem becomes more urgent when we want to know whether the stalk $\mathcal F_p$ at $p$, i.e. the colimit $\varinjlim\limits_{p \in U} \mathcal F(U)$, is again a member of $\mathsf{NiceCat}$. For all we know, it might be isomorphic to some ring of differentiable functions on an open set; proving that it isn't is going to be tedious and frustrating in any case.
Therefore, we decide to choose a suitable category (in casu, one with directed colimits) as the codomain of our (pre)sheaves, for example, $\mathsf{Ring}$.