Let's say we have two points on the unit sphere, $L_1$ and $L_2$ that are not poles. I want to express the isometry $I$ that rotates along the shorter line segment connecting them a certain number of degrees.
Essentially $d(I(L_1),L_1) = x$.
How does one do this? Is there a simple scaling factor in the rotation matrix? Do we just multiply the matrix by a constant (doubtful but believable)? I am really not sure.
What I am doing is creating an enemy in a video game that consists of balls connected by lines and so their motion is done by rotating around each other towards the player's direction. Hence, the isometry between their direction and the player's direction has to be scaled down to some basic velocity.
You are saying that your two points are neither identical nor antipodal. As vectors, they also have length one, and we can use the cross product and dot product.
First, the plane in which both lie is orthogonal to $$ U = \frac{L_1 \times L_2}{|L_1 \times L_2|}. $$ We are saying that $U \neq 0 $ and $|U|=1.$
There are a few ways to do this: we want a unit vector $V$ which is orthogonal to $L_1$ and lies in the $L_1 L_2$ plane, meanwhile is "on the same side" as $L_2.$ We use Gram-Schmidt
$$ V = \frac{L_2 - (L_1 \cdot L_2) L_1}{| L_2 - (L_1 \cdot L_2) L_1 |} $$ Note $$ V \cdot L_1 = 0, \; \; \; |V| = 1 $$
If you want a raw rotation, $$ L_1 \cos \theta + V \sin \theta $$
If you wanted a sort of prorated rotation, so that saying $90^\circ$ gives you $L_2,$ first find $$ \beta = \arccos L_1 \cdot L_2 $$ Given $\theta,$ take $$ L_1 \cos \left( \frac{\theta}{90^\circ} \beta \right) + V \sin \left( \frac{\theta}{90^\circ} \beta \right) $$ Here, when $\theta = 0,$ we get $L_1.$ When $\theta = 90^\circ,$ we get $$ L_1 \cos \beta + V \sin \beta = L_2 $$
It occurs to me to wonder how you use matrices. In the usual setup multiplying an orthogonal matrix times a column vector, let the three columns of $P$ be $U, L_1, V.$ The "destination" matrix $Q$ has columns $U, L_1 \cos \gamma + V \sin \gamma, - L_1 \sin \gamma + V \cos \gamma$ for angle $\gamma$ given by one of the schemes above. The the full rotation matrix $R$ is just $$ R = Q P^{-1} = Q P^T $$