I'm trying to understand the geometry of the Cayley transform. Suppose I have a $3 \times 3$ rotation matrix $R$ (i.e an orthogonal matrix with determinant equal to $1$). Let's ignore the corner case where $-1$ is an eigenvalue of $R$ (in other words, we assume that the rotation angle is not $\pi$). Then, according to a result of Cayley, I can find a skew symmetric matrix $S$ such that $$ R = (I - S)(I + S)^{-1} $$ In other words, I can find two other transformations $A = I - S$ and $B= (I + S)^{-1}$ whose combined effect, when applied one after the other, is the same as the original rotation.
My question is:
Can we find some geometric interpretation of the transforms $S$ and $A$ and $B$, so that we can see how they combine to produce a rotation.
I know that a rotation can be written as a product of two reflections. Is that related to the Cayley decomposition $R = AB$? Are $A$ and $B$ reflections?
The 3-dimensional case is the only one that's of interest to me.
Edit: Some Progress
I made some progress on the algebra, but not the geometry. Suppose our matrix $R$ corresponds to a rotation through an angle $\theta$ around the unit vector $\mathbf{n} = (u,v,w)$. Let $t = \tan\tfrac12\theta$. Then I managed to show that the Cayley decomposition is given by $R = A \cdot B$, where $$ S = \left[ \begin{matrix} 0 & t w & -t v \\ -t w & 0 & t u \\ t v & -t u & 0 \end{matrix} \right] $$ $$ A = I - S = \left[ \begin{matrix} 1 & -t w & t v \\ t w & 1 & -t u \\ -t v & t u & 1 \end{matrix} \right] $$ $$ B = (I + S)^{-1} = \frac{1}{1+t^2} \left[ \begin{matrix} t^2 u^2+1 & t (t u v-w) & t (v+t u w) \\ t (t u v+w) & t^2 v^2+1 & t (t v w-u) \\ t (t u w-v) & t (u+t v w) & t^2 w^2+1 \end{matrix} \right] $$ We have $\det(A) = 1+ t^2$ and $\det(B) = 1/(1+t^2)$, so neither $A$ nor $B$ is a rotation or a reflection.
I still don't see the geometry of $A$ and $B$, though. That's the puzzle.
Expanding on my comment ...
Write $$S = \left[\begin{matrix} 0 & r & -q \\ -r & 0 & p \\ q & -p & 0 \end{matrix}\right] \qquad M = I + S = \left[\begin{matrix} 1 & r & - q \\ -r & 1 & p \\ q & -p & 1 \end{matrix}\right] = ( I - S )^\top = N^\top$$
Note that $M$ (and $N$) fix the unit vector $\mathbf{p} := \frac{1}{s}(p,q,r)$, where $s^2 = p^2 + q^2 + r^2$.
Let $R$ be the reflection, through a plane containing the origin, that exchanges $\mathbf{z} := [0,0,1]^\top$ and $\mathbf{p}$. The normal to the plane is $\mathbf{z} - \mathbf{p}$, and we can compute the matrix as $$R = \frac{1}{s(r-s)}\left[\begin{matrix} r s - q^2 - r^2 & p q & p ( r - s ) \\ p q & r s - p^2 - r^2 & q ( r - s ) \\ p(r-s) & q ( r - s ) & r ( r - s ) \end{matrix}\right]$$
Then we have $$\widehat{M} := R^{-1} M R = R M R = \left[\begin{matrix} 1 & -s & 0 \\ s & 1 & 0 \\ 0 & 0 & 1 \end{matrix}\right]$$ such that $$\widehat{M} \left[\begin{matrix} a \cos\alpha \\ a \sin\alpha \\ b \end{matrix}\right] = \left[\begin{matrix} a ( \cos\alpha - s \sin\alpha ) \\ a ( s \cos\alpha + \sin\alpha ) \\ b \end{matrix}\right] = \left[\begin{matrix} a t \; \cos(\alpha+\theta) \\ a t \; \sin(\alpha+\theta) \\ b \end{matrix}\right] \qquad (*)$$ where $t^2 = 1 + s^2 = 1 + p^2 + q^2 + r^2$ and $\tan\theta = \frac{s}{1} = \sqrt{p^2+q^2+r^2}$.
The matrix $\widehat{M}$ represents the transformation that reflects the $z$-axis onto the fixed line of the transformation $M$, then applies transformation $M$, then reflects the fixed line back to the $z$-axis. As $(\star)$ indicates, if a point lies on a cylinder (of radius $a$) whose axis aligns with the $z$-axis, then the combined transformation moves that point to a $z$-aligned cylinder of radius $at$, but rotated by angle $\theta$ about the $z$-axis. Thus, $\widehat{M}$ ---and thus also $M$ itself--- could be called a "scaled rotation": it rotates points about its axis, and simultaneously scales the distances of points from that axis.