I have written an algorithm for solving the following problem: Given two 3d-vectors, say: $a,b$, find rotation of $a$ so that its orientation matches $b$.
However, I am not sure if the following algorithm works in all cases:
1) Find axis and angle using cross product and dot product:
$$\mathbf{x}=\frac{a\times b}{||a\times b||}\\ \theta=\cos^{-1}(\frac{a\cdot b}{||a||\cdot ||b||})$$
3) Find rotation matrix using exponential map:
$$\mathbf{R}=e^{\mathbf{A}\theta} =\mathbf{I}+\sin(\theta)\cdot \mathbf{A}+\left(1-\cos(\theta)\right) \cdot \mathbf{A}^{2}$$
where $\mathbf{A}$ is a skew-symmetric matrix corresponding to $\mathbf{x}$:
$$\mathbf{A}=[\mathbf{x}]_{\times}=\begin{bmatrix}0 & -\mathbf{x}_{3} & \mathbf{x}_{2} \\ \mathbf{x}_{3} & 0 & -\mathbf{x}_{1} \\ -\mathbf{x}_{2} & \mathbf{x}_{1} & 0\end{bmatrix}$$
Notes:
The axis is computed using cross product as this gives vector perpendicular to both $a$ and $b$. Only direction of the axis is important, hence it is divided by its magnitude. However, I am not sure if $\mathbf{x}$ will always have the proper direction (the result can be $-\mathbf{x}$ instead of $\mathbf{x}$?).
The rotation matrix is computed using Rodrigues' rotation formula.
Finally, the vector $\mathbf{R}a$ should have same direction as $b$.
I have tested this numerically and it seems working, but I would like to be sure the formulas work for any two $a,b$.
This is the right general approach, but the corner case $\|a\times b\| \approx 0$ must be handled.
If $\theta < \epsilon,$ $R=I$.
If $\pi-\theta < \epsilon$, you can choose for $\mathbf{x}$ any vector orthogonal to $\mathbf{a}$, for instance $\mathbf{x} = \frac{\mathbf{a} \times e_i}{\|\mathbf{a}\times e_i\|}$, where $i$ is the index of the component of $\mathbf{a}$ with least magnitude.