I wonder which characteristics of a (structured) set $X$ let us consider it a set of numbers.
Does the existence of two operations $+: X \times X \rightarrow X$ and $\times: X \times X \rightarrow X$ suffice which obey some laws, e.g. the laws of a ring? So would we consider each and every ring a set of "numbers"? Or has there to be more? Which structured sets are far from being a ring but considered a set of numbers nevertheless?
For example, the set of natural numbers doesn't form a ring, but is considered a set of numbers. And the set of prime numbers doesn't form a ring, but is considered a set of numbers. So maybe being a subset of a ring does suffice? But each and every subset? What are the minimal requirements?
Note, that there are different kinds of numbers:
counting (cardinal) numbers
ordering (ordinal) numbers
relating (rational) numbers
constructible (algebraic) numbers
measuring (real) numbers
Some kinds of numbers don't fit into this scheme, e.g. Gaussian integers, complex numbers or quaternions, or do they?
What do these kinds of numbers have in common?
[This question has to do with the question What is (a) geometry? which may be rephrased: What makes a (structured) set a set of points?]
There was a time when each of $-1, 0, $ and $1$ were not considered numbers.
The modern use of structures, $\{[\mathbf S, +, \times] + \text{axioms}\}$, for example, allows us to abstract away nonessential details and create useful theorems. Structures give names to a thing, but they are more concerned with proving theorems about that thing.
Feynman on the difference between knowing a thing and knowing about a thing
I think that, because of mathematical structures, the exact definition of number will always be a fuzzy and controversial thing. I don't plan on losing any sleep over that.