If I have two points $a$, $b$, on a unit sphere, I believe I can determine the angle between them, expressed as vectors, as follows:
$$\theta = \arccos\left(\frac{a\cdot b}{\|a\| \|b\|}\right)$$
However, as far as I understand it, this will always give a result $< 180$ degrees.
If I want to apply a rotation around the axis perpendicular to the plane defined by $a$, $b$, and origin, to transform point $a$ to point $b$, is there a method to determine the required direction of rotation (or direction along the axis, I suppose)?
If the axis of rotation is a constant, then at times I'll need to use $θ$ and at times I'll need to use $(360 - θ)$, depending on the values of $a$ and $b$.
As long as $a$ and $b$ are not proportional (i.e., distinct and not antipodal): If you rotate about an oriented axis, you always rotate by $\theta$ from $a$ to $b$ about the unit vector $$ n = \dfrac{a \times b}{\|a \times b\|}, $$ in the sense that $$ b = (\cos\theta)a + (\sin\theta)(n \times a). $$