In my molecular dynamics simulations, I have spherical particles with patchy surfaces (please see the figure) which means their relative orientation (kind of Euler angles?) is important in determining the interaction potential between each pair of particles (the potential has angular dependence). I use quaternion of the particles to get their orientation vector $u_{1,2}$ (which is correct). Based on this orientation, I will calculate the potential between each pair of particles. Initially, all the particles are aligned parallel to the $z$ axis of the reference (external) frame. In the attached figure, you can see how two of the three angles are defined based on the orientation vector of each particle. The trick to simplify the analytical calculation of the potential is that I assume that the line which connects pairs of particles is the local (body) $z'$ axis. Then I can define $\beta_1$ and $\beta_2$ as the angle between the orientation vector and the $z'$ axis. $\gamma_1$ and $\gamma_2$ are defined as the torsion (dihedral) angles which represent how the orientation vectors are twisted. The remaining angles, $\alpha_1$ and $\alpha_2$, that I am struggling with, are the rotation of each particle around its orientation vector.
Please see the figure; definition of the angles
Based on the previous posts on the web (Stack Exchange ...), I have tried using Euler angles based on the quaternion of the particles and rotate them according to the body $z'$ axis but was not successful. Then I have tried Rodrigues' rotation formula which was not successful as well.
When the surface patches are symmetric about the orientation vector, only beta and alpha are needed, the results are correct, but for general cases I do not get it right.
I think I have to find the third angle, $\alpha$, in a similar way that I have determined the $\beta$ and $\gamma$ angles, not by using Euler angles. Now I am really confused about any connection between the angles that I am using and the Euler angles that I can get from the quaternion of the particles.
Could you please let me know what should I do to solve this problem? How can I determine how much each particle is rotated about its own orientation vector? How can I correctly relate the angles that I am using to the Euler angles?
Thank you.
Edit 1.
It seems I have found kind of solution to the problem, but I should wait for the results of the simulation to see if it is working, so I highly appreciate your solutions/comments.
As I have the rotation matrices of the orientation vectors, I rotated the reference frame $x$-axis together with the the orientation vectors ${u_{1,2}}$. Now the angle between the rotated $x'$-axes ($x'_1$ and $x'_2$; body frame) shows how much the two particles are rotated (relative rotation: $\alpha_1-\alpha_2$) about their own orientation vectors.
I will update this post.