I have a matrix $M \in \mathbb{R}^{n \times m}, n \gt m \geq 3$, where every column of $M$ is mutually orthogonal (so rank($M$)=$m$).
I want to tranform $M$ in such a way that the span of $M$ is preserved. Intuitively, this is a rotation, but when I've investigated the question How i can rotate a $m$ dimensional vector? it makes it seem like "rotation" must have a different sort of meaning in greater than 3 dimensions than how I understand it (eg, if $n-m=1$, the solution doesn't help me because the "axis" of rotation needs to be $\mathbb{R}^{n-2 \times n}$?)
Basically, I want a family of matrices $R \in \mathbb{R}^{n \times n}$, such that span($RM$)=span($M$) for arbitrary dimension. How can I get that?
First of all, it may interest you to know that $\operatorname{span}(MR) = \operatorname{span}(M)$ as long as $R$ is invertible (of size $m \times m$).
To answer the question that you posed, namely find the family of matrices $R \in \mathbb{R}^{n \times n}$, such that span($RM$)=span($M$) for arbitrary dimension, here is a solution I like:
Let $M_0$ be the matrix whose columns are the normalized columns of the matrix $M$ (so that the columns of $M_0$ form an orthonormal basis of the column space of $M$). Extend these columns into an orthonormal basis of $\Bbb R^n$ (e.g. by the Gram Schmidt process) to obtain the columns of the matrix $$ \tilde M = \pmatrix{M_0 & M_1} $$ A matrix $R$ will preserve the span of $M$ if and only if it can be written in the form $$ R = \tilde M S \tilde M^T =\pmatrix{M_0 & M_1}\pmatrix{S_{11} & S_{12}\\0&S_{22}}\pmatrix{M_0 & M_1}^T $$ where $S$ is a block-matrix partitioned so that $S_{11}$ is an invertible $m \times m$ matrix and $S_{22}$ is $(n-m) \times (n-m)$.
To put it another way, $R$ will preserve the column space in question if and only if the matrix $$ S = \tilde M^T R M $$ is block-upper-triangular, i.e. partitioned in the manner described above, with $S_{11}$ invertible.
To put it yet another way, $R$ will preserve the column space in question if and only if we have $$ S_{12} = M_1^T R M_0 = 0 $$ and $S_{11} = M_0^TRM_0$ is invertible.