I am trying to let the user of my app rotate a $3D$ object drawn in the center of the screen by dragging their finger on screen. A horizontal movement on screen means rotation around a fixed Y axis, and a vertical movement means rotation around the X axis. The problem I am having is that if I just allow rotation around one axis the object rotates fine, but as soon as I introduce a second rotation the object doesn't rotate as expected.
Here is a picture of what is happening:

The blue axis represents my fixed axis. Picture the screen having this fixed blue axis. This is what I want the object to rotate in relation to. What is happening is in red.
Here's what I know:
The first rotation around $Y (0,1,0)$ causes the model to move from the blue space (call this space $A$) into another space (call this space B) Trying to rotate again using the vector $(0,1,0)$ rotates around the $x$ axis in space B NOT in space $A$ which is not what I mean to do.
Here's what I tried to fix this, given what I (think) I know (leaving out the W coord for brevity):
- First rotate around $Y (0, 1, 0)$ using a Quaternion.
- Convert the rotation Y Quaternion to a Matrix.
- Multiply the inverse of the $Y$ rotation matrix by my fixed axis $x$ Vector $(1, 0, 0)$ to get the fixed $X$ axis in relation to the new space.
- Rotate around this new X Vector using a Quaternion.
This isn't working how I expect. The rotation seems to work, but at some point horizontal movement doesn't rotate about the $Y$ axis, it appears to rotate about the $Z$ axis.
I'm not sure if my understanding is wrong, or if something else is causing a problem. I have some other transformations I'm doing to the object besides rotation. I move the object to the center before applying rotation. I rotate it using the matrix returned from my function above, then I translate it -2 in the Z direction so I can see the object.
I think your problem is related to frame of reference. Your global frame is A and other frames obtained after rotation in A are local frames (such as B). So u want all your rotations in global frame but they are taking place in local frame. For rotation in global frame of reference you need to pre-multiply the transformation matrix and for rotation in local frame, post-multiply. For eg, if u want rotation by $\theta_1$ in A for which transformation matrix is $R_{\theta_1}$ and then rotation by $\theta_2$ in A (rotation matrix $R_{\theta_2}$), then: $$V_f = R_{\theta_2}*R_{\theta_1}*V_i$$ where $V_i$ is initial position of the point(or object) and $V_f$ is final. for further reference, check this link: Maths - frame-of-reference for combining rotations