When I represent the orientations of 3D objects (with 3 degrees of freedom), can I embed them into R^N space in such a way that two orientations correspond to close vectors? So that the vector representation is at least locally smooth?
Like we do for 2D orientations: just take a complex number of unit length, and it nicely lays on the unit ring where close orientations correspond to close complex numbers.
Quarternions of opposite signs correspond to the same rotation [edited]
This may not exactly answer your question, but I've often used the Lie algebra associated with quaternions to deal with them in a local vector space (for state estimation purposes)
Let's say I have two quaternions $q_1$ and $q_2$.
$$\theta = 2\log(q_1^{-1} q_2) $$ $$ \theta \in \mathbb{R}^3 $$
Where the quaternion logarithm maps the the quaternion group to the associated lie algebra with:
$$ log(q) = \dfrac{q_v}{\lVert q_v \rVert} \textrm{atan2}(\lVert q_v \rVert, q_w)$$
and the quaternion exponential maps back to the Lie group
$$ \textrm{exp}(\theta) = \begin{bmatrix} \cos \lVert \theta \rVert \\ \textrm{sinc} \lVert \theta \rVert \theta \end{bmatrix} $$
The cool thing about this is that $\theta$ is a vector. You can add and subtract them locally, then map back to the quaternion space.
Hertzberg et al. defines new $\boxplus$ and $\boxminus$ operators which do this automatically, so you can almost pretend that quaternions are a vector space.
$$ q_1 \boxplus \theta = q_2 $$ $$ q_2 \boxminus q_1 = \theta $$
Depending on your application, you might be able to get away with just looking at the difference between quaternions in the algrebra ($\mathbb{R}^3$)