In an affine space, convex set $X$ is a set such that for each tuple of points $x,y\in X$ we have that $z=x+a\cdot(y-x)\in X$,(or in the case of a vector space, we can say $z=a\cdot x + (1-a)\cdot y \in X$) for $a\in (0,1)$.
But this definition of $z$ relies on the fact that in an affine space we can “move from $x$ in the direction of $y$”.
Yet it seems to me that we can also have a meaningful definition of convex set on e.g. a sphere (since we can move “in the direction of another point on the sphere” by moving along the arc on the sphere that contains both $x$ and $y$).
What is the most general space in which convexity can be meaningfully defined?
Asking for "the most general" definition is tricky, because as soon as a candidate a presented, a sufficiently motivated person can generalize it even further. That said: on a Riemannian manifold, or more generally on a geodesic metric space, one has the notion of Geodesic convexity: instead of asking for a set $X$ set to contain all segments with endpoints in $X$, one asks it to contain all geodesics with endpoints in $X$. This includes all the examples mentioned in the question.