Basic arithmetic teaches us the value of $1$ by counting (i.e. apples or oranges). A more advanced teaching of the count reveals that the count is a concept where each count of $1$ is exactly identical. This is different from the real world. Reapplying this means each apple we count is identical to all the other apples.
As I understand it the count would break down if we were counting varying things (i.e. $1$ apple, $2$ oranges, $3$ planets) yet we might say we have $1,2,3$ things. To explain that better I am saying I have $1$ orange $1$ apple and $1$ planet and the count is $1,2,3$ things.
Then when we advance a little further we begin to assign variables $x=1$ for example.
$x=1$ is interesting to me because if $x=1$ then does it not follow that $1=x$
If $1=x$ are we not at that point assigning a value of $x$ to $1$
Is the value of $1$ relative?
This is an interesting and confusing question, likely to be closed because it's not precise enough for an answer. That said, I will try to clarify some things.
When you use the numbers $1, 2, 3, \ldots$ to count, you can count whatever you like - all apples, or apples and oranges. Then you are doing "applied mathematics" and it's your responsibility to make clear what you are counting. But it's not the number $1$ that changes its meaning.
When you use the number $1$ in an equation like $x=1$ you are probably planning to do algebra. Then the $1$ is just one of the possible values for $x$, and might not be counting anything at all. But whatever it means there, $x=1$ and $1=x$ say the same thing, because the equal sign means precisely that the things on either side of it are the same object, perhaps named differently.
If you were writing a computer program rather than doing mathematics the equal sign might say "assign a value to a variable", and $1=x$ would be nonsense.
In arithmetic both the equations $$ 2 + 3 = 5 \text{ and } 5 = 2 + 3 $$ say the same thing. (Sometimes kids in the early grades don't think so, because the second one is not what they are usually asked about, and you can't use a calculator that way.)
Edit in response to comment.
You ask
As a mathematician, I look at the numbers as abstract things we're calling "$1$", "$2$", and so on. The collection of numbers has many beautiful properties. You can do arithmetic with numbers. Of course mathematicians created these mathematical abstractions to mirror the everyday behavior of numbers defined only informally. (Plato would argue that the numbers exist before mathematicians created them, but that's another discussion.)
So the number $1$ does not need any "reference point". It just is.
When you want to use numbers to model something in the everyday world - perhaps to count - then what you decide to "count as the reference point" depends on how you frame your problem. Sometimes you will want to count only apples. Sometimes you count fruit. In either case you can add or subtract quantities of whatever you are counting.
The numbers are just waiting there, unchanging, free of context, for you to use whenever and however it's convenient.