It may not be necessary to have a general definition, but it does not seem as if it should be difficult to come up with one. One requirement might be that the mean of a set of numbers must always be greater than or equal to the smallest number and less than or equal to the largest number. Another rule might be that if you replace a number with a larger one then the mean increases. This second requirement would rule out median and mode as types of mean. To include median and mode, we could relax the requirement to say that if a number is replaced by a larger number then the mean is greater than or equal to the previous mean. On the other hand, using this modification would have the minimum and the maximum as being types of mean, which may be okay, but seems a bit odd.
I just thought of another possible requirement. For arithmetic, geometric and harmonic means, you can take any k values and replace them with k copies of their mean. For example, the arithmetic mean for (2,4,6) is the same as for (3,3,6), replacing 2 and 4 with their arithmetic mean. This also works for minimum and maximum but not for median and mode.
Playing around with the requirements, it is not difficult to come up with infinitely many functions that meet all of them. Any continuous increasing or decreasing function f can be turned into a mean function m(x) over a list of numbers x, using m(x)=f-1(arithmetic mean of f(x)). For arithmetic mean f is the identity function, for geometric mean f is log(x) and for harmonic mean f is 1/x.
I looked at the Web site that Qiaochu Yuan pointed to, and I see that what I was referring to in my previous paragraph is a quasi-arithmetic mean. There is a property that all the all the above candidates for a mean function have and which seems should be necessary. The mean function should be scale-free. Multiplying all the values by a constant should multiply the mean by that constant. It should not matter whether time values, for example, are given in hours or minutes. The mean should be the same time interval. According to the following Web site in the section on homogeneity, quasi-arithmetic means do not usually have the scale-free property. https://everipedia.org/wiki/lang_en/Quasi-arithmetic_mean
The Wikipedia page for the quasi-arithmetic mean (for a more precise statement see the original theorem in Hardy-Littlewood-Polya's Book "Inequalities" quoted on Wikipedia) states that the only quasi-arithmetic means that are "scale-free" are the ones where $f$ is on of the following functions $$ \label{eq:star} \tag{$\star$} f_r \colon (0, \infty) \to \mathbb R, \qquad x \mapsto \begin{cases} x^r, & \text{if } r \in \mathbb{R} \setminus \{ 0 \}, \\ \ln(x), & \text{if } r = 0. \end{cases} $$ On that Wikipedia page you can also read that the requirements on $m$ which you state in the first paragraph of your question nearly already imply that $m$ must be an quasi-arithmetic mean (for proofs look at the relatively short papers by Kolmogorov or Nagumo cited there).
In short, the family of means is exactly the family of power means \eqref{eq:star} described above.
I hope this helps, if not, please comment.