I am looking for ways to parametrize 3D rotations of an object.
I am thinking of using the roll yaw and pitch angles but I could not find formulas that express what is the resulting roll yaw and pitch obtained from performing two consecutive rotations with known roll yaw pitch angles.
Do such formulas exist? if not what prevents their existence?
Rotation of $\phi$ about some axis $w$ can be represented by a matrix denoted by $R_{w}(\phi)$. Then roll, pitch and yaw angles, denoted by $\phi_r, \phi_p$ and $\phi_y$ respectively, correspond to the rotations, given in [1]: \begin{align*} R_{z}(\phi_r) &= \begin{pmatrix} \cos{\phi_r} & -\sin{\phi_r} & 0\\ \sin{\phi_r} & \cos{\phi_r} & 0\\ 0 & 0 & 1 \end{pmatrix} \\[.5em] R_{y}(\phi_p) &= \begin{pmatrix} \cos{\phi_p} & 0 & \sin{\phi_p}\\ 0 & 1 & 0\\ -\sin{\phi_p} & 0 & \cos{\phi_p} \end{pmatrix} \\[.5em] R_{x}(\phi_y) &= \begin{pmatrix} 1 & 0 & 0\\ 0 & \cos{\phi_y} & -\sin{\phi_y}\\ 0 & \sin{\phi_y} & \cos{\phi_y} \end{pmatrix} \end{align*}
Given roll, pitch and yaw angles, the corresponding rotation can then be represented by a rotation matrix: \begin{equation*} R(\phi_r, \phi_p, \phi_y) = R_z(\phi_r)\, R_y(\phi_p)\, R_x(\phi_y) \tag{1} \end{equation*}
Performing two consecutive rotations with known roll, pitch and yaw angles $\phi_{*}'$ and $\phi_{*}''$, result in a matrix: \begin{equation*} R(\phi_r, \phi_p, \phi_y) = R(\phi_r', \phi_p', \phi_y') \, R(\phi_r'', \phi_p'', \phi_y'') \tag{2} \end{equation*}
The entries of matrices on the left hand side and right hand side in the above expression have to be equal. For example, expanding out and comparing the entries $(3, 3)$, we get an equation: \begin{align*} \cos{\phi_y} \cos{\phi_p} &= \cos{\phi_y'} \cos{\phi_y''} \cos{\phi_p'} \cos{\phi_p''} \\ &\quad- \cos{\phi_p'} \cos{\phi_r''} \sin{\phi_y'} \sin{\phi_y''} \\ &\quad- \cos{\phi_y''} \cos{\phi_r''} \sin{\phi_p'} \sin{\phi_p''} \\ &\quad- \sin{\phi_y''} \sin{\phi_p'} \sin{\phi_r''} \\ &\quad+ \cos{\phi_y''} \cos{\phi_p'} \sin{\phi_y'} \sin{\phi_p''} \sin{\phi_r''} \end{align*}
Solving these equations for $\phi_{r}, \phi_{p}, \phi_{y}$ is in general hard. We can get different sets of solutions depending on the range we assume for $\phi_p$. Also, when $\phi_p'$ and $\phi_p''$ are equal to $\pi/2$ or $-\pi/2$, we lose one degree of freedom, this is so called gimbal lock, and we will not be able to solve for $\phi_p$.
To parameterize rotations, you can just use rotation matrices. Another popular possibility is to use unit quaternions embedded in $\mathbb{R}^{4}$. Then the composition of rotations corresponds to the product of the corresponding quaternions. Also, the quaternions remain singularity-free which allows you to avoid gimbal lock.
Remark. There is no simple rule for $\phi_{*}$ in terms of $\phi_{*}'$ and $\phi_{*}''$ if we choose to define each consecutive rotation $R$ as defined in Eq. (1) for given angles $\phi_{*}'$ and $\phi_{*}''$, for $* \in \lbrace r, p, y \rbrace$.
One could find simple rules for special cases. For example, say we are given angles $\phi_{r}', \phi_{r}''$ to perform only two consecutive rolls. Then Eq. (2) becomes $$ \begin{pmatrix} \cos{\phi_r} & -\sin{\phi_r} & 0\\ \sin{\phi_r} & \cos{\phi_r} & 0\\ 0 & 0 & 1 \end{pmatrix} = \begin{pmatrix} \cos{(\phi_{r}' + \phi_{r}'')} & -\sin{(\phi_{r}' + \phi_{r}'')} & 0\\ \sin{(\phi_{r}' + \phi_{r}'')} & \cos{(\phi_{r}' + \phi_{r}'')} & 0\\ 0 & 0 & 1 \end{pmatrix} $$ implying a simple rule $$ \phi_{r} = \phi_{r}' + \phi_{r}'' $$ similarly for pitch or yaw.
[1]: https://en.wikipedia.org/wiki/Rotation_matrix