I am reading this blog about positional encoding.
I came across an interesting comparison where the continuous representation of bit change is a sinusoidal function.
See the following bit change:
0 -> 0 0 0 0
1 -> 0 0 0 1
2 -> 0 0 1 0
3 -> 0 0 1 1
4 -> 0 1 0 0
5 -> 0 1 0 1
You can spot the rate of change between different bits. The LSB bit is alternating on every number, the second-lowest bit is rotating on every two numbers, and so on.
But using binary values would be a waste of space in the world of floats. So instead, we can use their float continuous counterparts - Sinusoidal functions. Indeed, they are the equivalent to alternating bits. Moreover, By decreasing their frequencies, we can go from LSB bits to MSB