Let $M$ be a compact 2-dimensional smooth Riemannian manifold of genus $g$ and $S\subset M$ be a finite set. We define a barycenter of $S$ to be a point $p\in M$ such that
$$\inf_{y\in M} \sum_{x\in S} d(x,y)=\sum_{x\in S} d(x,p)$$
where $d$ is the Riemannian distance. Is it true that if $S$ is not contained in $g+1$ geodesics, then the barycenter is unique?
This conjecture is motivated by the following counterexamples:
If $S$ is a set of equally spaced points on the equator of a sphere, then the south and north poles are barycenters.
If $M$ is the torus obtained by identifying the sides of the square $[0,1)^2$ and $S=\left\{0,\frac 12\right\}\times\left\{0,\frac 1n,\dots, \frac{n-1}n\right\}$ then the barycenter is also not unique.