Suppose that I have two 3D triangles sharing an edge, as shown by the picture below.
The Two triangles, i.e. $\Delta ABC$ and $\Delta BCD$ share the edge $BC$. The coordinates of nodes $A, B, C$ and $D$ are known.
Now, I want to rotate $\Delta ABC$ to $\Delta BCD$ around $BC$, making the two triangles partially overlapping (because two triangles are different).
By the way, I have to make sure that, after rotation, the two triangles partially overlap, instead of being adjacent on the same 3D plane.
The quaternion might be a solution, but I kind of dislike it, because the direction of rotation axis could affect the result, i.e. $BC$ or $CB$. Also, to use the quaternion rotation, the angle between the two triangles is required.
Is there any sound solution just based on the node coordinates?
You can choose the axis to be
$\vec v = (C-B)$
then you can define a vector in the plane of $\Delta ABC$ as
$\vec a = \vec v \times \left( (A-B)\times \vec v \right)$
likewise for $\Delta ADC$
$\vec d = \vec v \times \left( (D-B)\times \vec v \right)$
Then the angle between them would be
$\theta = \cos^-1 \left( \frac{\vec a \cdot \vec d}{| \vec a |\cdot | \vec d|} \right)$
this represents the rotation in terms of the axis $\vec v$ and the angle $\theta$.
You can then use documentation on the axis-angle representation to convert to a rotation matrix using this site: https://en.wikipedia.org/wiki/Axis%E2%80%93angle_representation
Once you have the rotation matrix $R$ you can transform the point $A$ with
$A' = R(A-B) + B$
Note that there are cleaner ways to do this but this is the most intuative.