Background:
I am interested in extending what I'm calling the "great circle" principle - for the shortest path along the surface of a sphere - to apply to rotations in three dimensions, i.e. for finding shortest paths between matrices in SO(3). What I mean by this is that when constrained to the surface of a sphere, the shortest distance between any two points lies along a circle which bisects the sphere and contains the two points in its circumference.
When considering rotations of a rigid body, this type of knowledge is useful for finding the shortest path between two orientations, but only when considering two rotational degrees of freedom. Given two vectors (say, a starting and a final pointing vector for the rigid body), one can find the angle between them via the dot product, and an axis about which to rotate (via the cross product) that will generate the shortest path between them. These vectors, however, can't encode any information about the third angle of the orientation.
Based on Euler's theorem, I think intuitively that the axis-angle representation of the relative rotation between two orientations should give me the same effect for arbitrary three-dimensional rotations.
Question(s):
1) For the strictly rotational three-dimensional case, is the relative axis-angle rotation the shortest path between two orientations? Based on my spotty understanding of Euler's rotation theorem, it's well known that only that any rotation can be represented by one angle and an axis, not necessarily that it's the shortest path.
While researching this question, I found this article: https://en.wikipedia.org/wiki/Geodesic. This seems to describe the process by which one might prove this, but calculus of variations is (currently) a bit beyond my grasp.
2) In a similar vein as the geometric description I gave above, could you interpret the shortest path between two orientations as some portion of a sphere which intersected with a four-dimensional hypersphere?
Yes, it's the shortest path.
Why? Because there's an nice covering $f$ of $SO(3)$ (the space of 3x3 rotation matrices) by $S^3$ (the 3-sphere in 4-space). "Nice" in this case means two things:
It's a 2-to-1 map. If $f(P) = R$ for some point $P$ and some rotation $R$, then $f(-P) = R$ too, and $f(X) = R$ ONLY for $X = P, -P$.
It locally preserves geometry, i.e., if $P$ and $Q$ are sufficiently close, then the shortest path from $P$ to $Q$, when transformed by $f$, gives you the shortest path from $f(P)$ to $f(Q)$.
You can examine a very similar map by considering the map $$g: S^1 \to S^1 : \theta \mapsto 2\theta.$$
That one's also 2-to-1, and preserves shortest paths...as long as they're not too long.
The good thing about the map $f$ is that we definitely know what the geodesics on $S^3$ are -- they're great circle arcs. And we also know (courtesy of Rodrigues' formula for the map $f$ -- see wikipedia, or lots of answers here on MSE) that such geodesics transform exactly into paths of the form $$ \gamma(t) = Rot(v, t) $$ where $0 \le t \le a$ is an interval, and $v$ is a fixed unit vector, and $Rot(v, t)$ means "rotate about $v$ by angle $t$."
For the second question -- -I can't quite make sense of what you wrote. But the arc on $S^3$ corresponding to a geodesic in rotation space is a great-arc, hence it lies on the intersection of $S^3$ with some 2-plane through the origin in $4$-space, which might be what you were asking.