I need to figure out how to turn two rotation groups, each rotating around Z, X then Y into a single rotation group, so that in one set of rotations I might obtain the same positions for a set of rotated points as after two consecutive sets of ZXY rotations.
So for a rotation group given by
$$ R_{zxy}(\psi, \varphi, \theta) = \begin{bmatrix} \cos \psi & -\sin \psi & 0 \\ \sin \psi & \cos \psi & 0\\ 0 & 0 & 1\\ \end{bmatrix} \begin{bmatrix} 1 & 0 & 0 \\ 0 & \cos \varphi & -\sin \varphi \\ 0 & \sin \varphi & \cos \varphi \\ \end{bmatrix} \begin{bmatrix} \cos \theta & 0 & \sin \theta \\ 0 & 1 & 0 \\ -\sin \theta & 0 & \cos \theta \\ \end{bmatrix} $$
I need to find $\psi_3, \varphi_3, \theta_3$ from $\psi_1, \varphi_1, \theta_1$ and $\psi_2, \varphi_2, \theta_2$ so that for a point P
$$ P' = P R_{zxy}(\psi_3, \varphi_3, \theta_3) = P R_{zxy}(\psi_1, \varphi_1, \theta_1) R_{zxy}(\psi_2, \varphi_2, \theta_2) $$
Euler angles are not a good representation for multiplying rotations. In principle, it would be possible to write explicit formulas for the function you want, but the easier way to do this is to use the standard conversion between Euler angles and one of the other representations that are better at dealing with products, e.g. rotation matrices or unit quaternions. You can transform your two sets of Euler angles to one of those representations, do the multiplication there, and then transform back. If you really need a functional form, e.g. to take derivatives, you can view the whole process as chaining together three functions (transform, multiplication, inverse transform).