I'm looking for a sensible way to parametrize the full set of possible 3D rotations that satisfy the property that they don't rotate any vector in $\mathbb{R}^3$ by more than some given angle $\alpha$, where $\alpha < \pi/2$.
If we were to use rotation matrices to describe the rotation, then the set can be perhaps(?) defined as:
$$S = \{R \in SO(3) \mid \max\{\cos^{-1}(e_1^TRe_1), \cos^{-1}(e_2^TRe_2), \cos^{-1}(e_3^TRe_3)\} < \alpha\},$$ where ${e_i}$ are the canonical basis vectors - though I'm not at all sure if these three vectors provide a sufficient constraint.
I'm struggling to rephrase the initial constraint on $\alpha$ into constraints on the rotation parameters such as Euler angles, or quaternions for non-trivial rotations.
The goal is to be able to sample and apply these rotations randomly, but also to understand what the above property means analytically.