I'm not sure if I'm using the right terminology on my title but I'll explain here. By "continuous representation space" I mean that I want to represent all physical rotations (call this the set $X$) in some manifold in $R = \mathbb{R}^n$ where I'm not so concerned about what the $n$ is. The conditions on this space would be:
- I have a mapping $g: X \rightarrow R$ and a mapping $f: R \rightarrow X$ and one mapping is the inverse of another. I think this is called a bijection.
- Small nudges to some $\mathbf{r} \in R$ cause proportionally small nudges to $\mathbf{x} = f(\mathbf{r}) \in X$. Same goes for the inverse. This is true everywhere.
It's known that the group of orthonormal matrices in $SO(3)$ are such a representation. Quaternions are not because they are a double cover on $SO(3)$ and even if you restrict them to one half space they still have a discontinuity with angles of $\pi$.
So far so good. But what if I wish to represent physical orientations of a perfect cube where there are 24 symmetries? This doesn't work with the $SO(3)$ matrices. For example a rotation of $\pi/4$ about one of the edges of the cube gives the same results as a rotation of $-\pi/4$ about that same edge. This situatuion satisfies nor 1 nor 2. To satisfy 1 I could squish $SO(3)$ up to only include those rotations that have the smallest geodesic distance to one of the 24 cube symmetries but this still wouldn't satisfy 2 (and I couldn't think of a way to write that squishing operation into closed form).
For reference/visualization, this GIF shows what happens when a take a small arrow and apply a uniform(ish) grid of all possible rotations to it. Then what happens if I remap all arrows via my squishing algorithm.
It sounds to me like you're looking for a description of the quotient space of $SO(3)$ by the subgroup of rotations of a cube, specifically an embedding of this space into Euclidean space. This is a homogeneous space for a Lie group (either $SO(3)$ or $SU(2)$) and lots of stuff is known about these.
This particular homogeneous space is, among other things, a closed smooth $3$-manifold. By the Whitney embedding theorem this implies that it can be smoothly embedded into $\mathbb{R}^6$. Hirsch, Wall, and Rokhlin apparently proved that a smooth $3$-manifold in fact embeds into $\mathbb{R}^5$. Unfortunately these proofs are, as far as I know, pretty non-explicit.
Interestingly this question has been asked before (with no answers), $6$ years ago, but that time the asker wanted a stronger condition of an isometric embedding. The space you're interested in can be described as a quotient of the $3$-sphere $S^3 \cong SU(2)$ by the binary octahedral group, which reveals it to be a spherical $3$-manifold. In particular it inherits (up to scale) a metric of constant positive curvature from the $3$-sphere and one can ask for an embedding which respects this metric (which, for example, is true of the embedding of $SO(3)$ into $\mathbb{R}^9$ given by the usual matrix representation). For a closed Riemannian $3$-manifold the Nash embedding theorem guarantees such an embedding into $\mathbb{R}^{12}$ but is again, as far as I know, pretty non-explicit.
Here's an idea for writing down a (terrible) explicit embedding which doesn't quite work, which is a variation of Yuval Peres' answer to your other question. We'll think of your space as the quotient of $S^3$ by the binary octahedral group which we'll just write as $\Gamma$. Write $x_1, \dots x_4$ for the $4$ coordinates of the usual embedding $S^3 \to \mathbb{R}^4$. These coordinates, thought of as functions on $S^3$, are acted on linearly by $\Gamma$. Now consider the $48$ non-leading coefficients of the $4$ polynomials
$$f_i(t) = \prod_{g \in \Gamma} (t - g x_i).$$
The coefficients are by construction smooth functions invariant under $\Gamma$, and moreover by construction knowing these coefficients is enough to recover the orbit $\{ gx_i \}$ of each coordinate function $x_i$ independently. That means if we know the values of all of these coefficients we know, generically, that they came from $48^4$ possible points, where each factor of $24$ comes from using one of each of the possible values of $\{ gx_i \}$ in each orbit. This is bad; what we'd like to do is to recover a single orbit of $48$ points. However, we can hope that most of the resulting $4$-tuples of vectors are not unit vectors (which they would need to be to lie on the $3$-sphere $S^3$) and so we can hope that in practice actually most of them are ruled out. I don't know if this happens but even if it does the resulting map takes values in $\mathbb{R}^{192}$ and is annoying to invert. We can probably remove the ambiguity entirely by adding a few more functions but the result would still be annoying to invert.
Generally speaking, pure mathematicians prefer to avoid working with embeddings at all and will just work directly on a manifold itself, independent of a choice of embedding, partly to avoid annoying issues like this. If you could say more about what you need computationally from such an embedding it might be easier to suggest alternative approaches.