Given a rotation matrix $Q \in \mathbb R^3$, how to find a magnitude of rotation. I guess it makes sense to say that identity matrix $I$ is considered 0 magnitude and the rotation of $-I$ is considered maximum magnitude. It would also makes sense that $mag(Q) = mag(Q^T)$.
Is there a way of computing this without going through the trouble of finding Euler angles and then deriving metric using identity $\cos^2(\alpha) + \cos^2(\beta) + \cos^2(\gamma) = 1$?
On
You don't really define what is meant by magnitude and I don't think that the Euler angles representation is appropriate for that purpose.
I suggest to use the axis/angle representation, and this method: https://en.wikipedia.org/wiki/Rotation_matrix#Determining_the_angle.
I found it useful to think about the transformation rotation matrix $Q \in R^{3 \times 3}$ affects given vector $v$. That means what does $$\vec u = \vec v \times Q^T $$ look like. After that I can compare the initial vector $\vec v$ and compare it to $\vec u$ by dot product between them since $$cos\ \phi = \frac{\vec u \cdotp \vec v}{||\vec u|| \ ||\vec v||}$$.
$ \vec v \in R^3$; $\vec v$ should be unit vector, let us choose $[1, 0, 0]$. giving the metric from $[-1; 1]$ of $$metric(Q) = (\vec v Q^T) \cdot \vec v;\ v = [1, 0, 0]$$
This simplifies to $metric(Q) = Q_{11}$, which agrees with maximum of 1, when $Q = I$, minimum of -1 when $Q = -I$ and $metric(Q) = metric(Q^T)$