I am reading some books and random online notes on Lie theory and Algebra. The terminalogy comes up alot
"Let $G$ be a (compact Lie) group that acts orthogonally on $\mathbb{R}^n$.."
A typical example would be from Bredon's "Introduction to Compact Transformation Groups (page 84)" when he discusses the existence of tubes about the orbit of a completely regular $G-$space.
If I dug out more of my random notes, I will also link a reference.
I am guessing this means that it carries orthogonal elements into its orthogonal space?
The orthogonal group $O_n$ is the group of $x \in \textrm{GL}_n(\mathbb{R})$ such that $\langle xv,xw \rangle = \langle v,w\rangle$ for all $v, w \in \mathbb{R}^n$, where $\langle - , - \rangle$ is the standard inner product. So in saying that a group $G$ acts orthogonally on $\mathbb{R}^n$, my guess would be this means that there is a group homomorphism of $\pi: G \rightarrow O_n$ through which the action of $G$ on $\mathbb{R}^n$ is defined.
This is of course under the assumption that the given group action of $G$ on $\mathbb{R}^n$ satisfies some basic properties: $g \cdot (v+w) = g \cdot v +g \cdot w$ and $g \cdot \lambda v = \lambda(g \cdot v)$ for $g \in G, v, w \in V, \lambda \in \mathbb{R}$.