Find an angle to rotate a vector around a ray so that the vector gets as close as possible to another vector.

Question

I have two Vectors: the Vector I need to move (Vector A) and the Vector I need to get as close as possible to (Vector B). I also have a Ray R (an origin + axis/direction) which Vector A should be rotated around. How do I find an Angle, that will bring Vector A as close as possible to Vector B, when rotating around the Ray R?

CONTEXT

I am building an editor extension for Unity, a custom gizmo for various transformations of objects: translations/rotations/scales. Among other things it should allow for "vertex-snap rotations" In the picture above, the tool should be able to rotate the cube in such a way around the custom axis, that Vector A and Vector B get aligned. This case is an easy one, because the axis is the Wold Y axis, I need a general solution for any axis and any kind of vectors.

The reason why I might have sounded confusing is because my tool allows to set next to any constrains on what rotations should be performed: the user can turn off X-axis rotation (to only rotate in YZ-plane), they can turn off two axes and even do the same for the object's local space. I've got a method that rotates the cube around an axis with an angle, all I need is to find the closest angle BUT for a specific axis.

Answer 1

$$ \newcommand{\va}{{\mathbf a}} \newcommand{\vv}{{\mathbf v}} \newcommand{\vb}{{\mathbf b}} \newcommand{\vu}{{\mathbf u}} \newcommand{\vf}{{\mathbf f}} \newcommand{\vc}{{\mathbf c}}\newcommand{\ve}{{\mathbf e}}$$

Let's suppose that $\va$ and $\vb$ are the vectors you care about, and that each is a unit vector, i.e. $\| \va \| = \| \vb \| = 1$. Let's suppose that $\vu$ is similarly a unit vector pointing along the axis of your ray.

I'm going to assume that $\va$ and $\vu$ are not parallel, for if they are, no amount of rotation about $\vu$ will change $\va$, so there's no "best angle."

OK then. Let $$ \vc = \va - (\va \cdot \vu) \vu \\ \ve = \frac{1}{\| \vc \|} \vc. $$ Then $\ve$ is a unit vector perpendicular to $\vu$ in the $\vu\va$-plane.

Let $$ \vf = \vu \times \ve $$ as well. Then let $$ \theta = \text{atan2} (\vb \cdot \vf, \vb \cdot \ve) $$ Then you want to rotate $\va$ about $\vu$ by angle $\theta$, where the sign depends on how you describe rotations about an axis (there are two reasonable choices, and they require the two different choices of $\pm$).

If you want the actual matrix of the rotation (assuming you're using column vectors, and transformations that look like $\vv \mapsto M\vv$), then let $$ A = \pmatrix{ \ve & \vf & \vu} $$ where that means that the columns of $A$ are the vectors $\ve,\vf,$ and $\vu$, and let $$ B = \pmatrix { c & -s & 0 \\ s & c & 0 \\ 0 & 0 & 1}, $$ where $c = \cos(\theta), s = \sin(\theta)$. Finally, let $$ M = A B A^t. $$

Answer 2

You could consider the two planes:

1. $\pi_A$ containing $A+RayR$

2. $\pi_B$ containing $B+RayR$

and then determine the angle between them.

I'm assuming that the distance between $A$ and $B$ is at a minimum when $\pi_1 \equiv \pi_2$.

Here is a sketch: