I've got the rotation matrix and quaternion of a mobile device. I'm trying to calculate the vertical angle from it. What I'm trying to get is for example 0° if the device is held in portrait mode and 90° in landscape mode.
Any help would be appreciated. I'm sorry if this was asked before, I don't have any experience with this and didn't really know what to search for.
There are many sign conventions associated with rotation matrices, quaternions, and Euler angles. It can get confusing.
But hopefully you just want the angle $\theta$ between the space-fixed $z$-axis and the body-fixed $z'$-axis, and presumably your $3\times3$ rotation matrix $\mathbf{A}$ is defined such that $\mathbf{r}'=\mathbf{A}\cdot\mathbf{r}$ (or the same equation, but with the transpose of $\mathbf{A}$). In that case $\cos\theta=A_{33}$ because it is just the scalar product between the original, and rotated, $(0,0,1)$ vectors. So take the arccos of the $3,3$ element of the rotation matrix.
I advise you to double check that this makes sense for your definitions and coordinate systems!