I am sorry if this has been answered and I am not being able to find.
I have a triangle with vertices at $A(x_1, y_1, z_1), B(x_2,y_2,z_2)$ and $O(0,0,0)$. Now, the triangle is rotated about the fixed origin to another position $A'(x_1', y_1', z_1'), B'(x_2',y_2',z_2'), O(0,0,0)$.
with my little knowledge of Euler's rotation theorem, I think there should be a unique rotation matrix to map these two positions of the triangle but don't know how exactly to calculate the matrix.
Any help with an efficient algorithm to calculate the rotation matrix would be very very helpful.
Assuming that these triangles aren’t degenerate, the vectors $A$, $B$ and $A\times B$ form a basis for $\mathbb R^3$, as do $A'$, $B'$ and $A'\times B'$. The rotation that takes the first triangle to the second also maps $A\times B$ to $A'\times B'$, so the matrix that maps the first basis onto the second will be the required rotation. The most straightforward way to find this matrix is to compute $$\begin{bmatrix} A' & B' & A'\times B'\end{bmatrix} \begin{bmatrix} A&B&A\times B \end{bmatrix}^{-1},$$ but this might not be the most efficient method.
Another way to go is to take advantage of the fact that the rotation axis must lie on the angle bisector of a point and its image. Thus, the required rotation axis is the intersection of the planes through the origin with normals $A'-A$ and $B'-B$. This line is perpendicular to both normals, so assuming that neither is zero (in which case the rotation is about that side of the triangle), this means that the rotation axis is parallel to $(A'-A)\times(B'-B)$. You can work out the rotation angle $\theta$ by using the inner product identity $\mathbf v\cdot \mathbf w = \|\mathbf v\|\|\mathbf w\|\cos\theta$ (in fact, you really only need $\cos\theta$ and not the angle itself) and then use Rodrigues’ formula.