Distance to median vs average intra-distances

Question

Consider $n$ points in a vector space, denoted $(a_1, \dotsc, a_n)$. I am wondering if the following inequality holds true: $$ \min_x \sum_{i=1}^n d(a_i, x) \leq \frac{1}{n-1} \sum_{i=1}^n \sum_{j \neq i} d(a_i, a_j) $$ where $d(.,.)$ is a distance (or a pseudo-distance) and $x$ can be any distance in the vector space. I tried to prove it in various ways but could not succeed. I could check it works in some cases, so I would be glad to see if it works in all cases or to get a counterexample!