$a,u,v \in \mathbb{R}^3$ have $a^Tu= 0$. If $R(a,\theta)$ is the rotation of space CCW $\theta$ radians about $a$, then what $\theta$ solve $v^TR(a,\theta)u=0$?
I believe any 2 answers have expressions in terms of a 2-argument arctangent but I have not been able to find a convenient reduced formula. A symbolic solver reports a large expression for both arguments of the arctangent.
From the Rodrigues' rotation matrix formula, we have
$R(a, \theta) = {a a}^T + (I - {a a} ^T ) \cos \theta + S_a \sin \theta $
where $S_a$ is the skew-symmetric matrix that satisfies $S_a x = a \times x$ for any vector $x$. That is,
$S_a = \begin{bmatrix} 0 && - a_z && a_y \\ a_z && 0 && - a_x \\ -a_y && a_x && 0 \end{bmatrix} $
Now the condition becomes
$ v^T R(a, \theta) u = v^T {a a}^T u + v^T(I - {aa}^T) u \cos \theta + v^T S_a u \sin \theta = 0 $
which is of the form
$ A \cos \theta + B \sin \theta = C $
with $A = v^T (I - {aa}^T) u , B = v^T S_a u , C = - v^T {a a}^T u $
And this problem can have 0, 1, or 2 solutions depending the value of $\dfrac{C}{\sqrt{A^2 + B^2}} $
Let $ \phi = \text{Atan2}(A, B) $
(i.e. $\cos \phi = \dfrac{A}{\sqrt{A^2 + B^2} } $, and $\sin \phi = \dfrac{B}{\sqrt{A^2 + B^2} } $), then
$ \cos(\theta - \phi) = \dfrac{C}{\sqrt{A^2 + B^2} } $
from which
$ \theta = \phi \pm \cos^{-1} \left( \dfrac{C}{\sqrt{ A^2 + B^2 } } \right) $
So, we will have no solutions if
$ \left| \dfrac{C}{\sqrt{A^2 + B^2}} \right| > 1 $
and one solution if
$ \left| \dfrac{C}{\sqrt{A^2 + B^2}} \right| = 1 $
and two solutions if
$ \left| \dfrac{C}{\sqrt{A^2 + B^2}} \right| < 1 $
In this problem, it is given that $a^T u = 0 $, hence
$A = v^T u , B = v^T S_a u , C = 0 $
Since $C = 0$ , we have two solutions,
$ \theta = \phi \pm \dfrac{\pi}{2} $