I'm trying to find a word or definition to capture something I've seen in theoretical physics. Physicists work with coordinate systems a lot, and they frequently represent physical facts as restrictions on what coordinate systems are valid, particularly by specifying transformations that preserve the set of valid coordinate systems (which I'm calling symmetries).
Basically, whenever we're talking about linear coordinate systems, we essentially have a vector space with some restricted set of symmetries. For instance, in Newtonian Mechanics, we model space with $\mathbb{R}^{3}$, except only orthonormal bases give valid coordinate systems. This is representing that the physics doesn't change if you rotate your coordinate system, but if you scale one of the axes (basis vectors) or apply a shear, it does change the physics. Put another way, we are modeling space with $\mathbb{R}^{3}$ equipped with the orthogonal matrices as our symmetry group instead of the general linear group (if you want to be very precise, we are actually modeling space/time with $\mathbb{R}^{4}$ and Galilean transformations).
It gets a bit more complicated in Special Relativity, where we model space-time as $\mathbb{R}^{4}$, but instead of the orthogonal matrices we have the Lorentz transformations. This amazingly encodes the fundamental ideas of the theory - that is, Special Relativity could be summed up by saying that spacetime is represented as a Minkowski space (sorta just $\mathbb{R}^{4}$ with Lorentz transformations).
Thus we get to my question: Is there a name for this idea or a mathematical definition for this sort of structure? I'm not sure there is really something generalizable here, so it might be a stretch. This might also be better described in terms of manifolds than linear spaces.