I have a rotation matrix created by some values of roll, pitch, yaw and I'm looking for the value yaw2 s.t the rotation matrix created by [0, 0, yaw2] is closest to the roration matrix created by [roll, pitch, yaw].
I think It can be rephrased as projecting the rotation matrix to the horizontal plane.
I'm defining my rotation matrix as Ryaw X Rpitch X Rroll, but does the answer change if I reverse the order?
After thinking and researching the problem further, the solution is to project the perpendicular vector.
e.g when the Yaw is the only rotation that is not zero, the up vector remains pointing at (0,0,1), regardless of the yaw value.
When taking the full rotation matrix, calculate the direction of the "up vector" by rotating it using the rotation matrix, and then find the rotation matrix that will rotate this new up vector back to (0, 0, 1). This is a common and easy problem, easiest using quaternions.
The multiplication of the original rotation matrix, with the new rotation matrix to align the up-vector back to its position will generate the rotation matrix. the last step is to extract the yaw value, this is easy as the rotation matrix is constructed only using sin,cos of the yaw