I'm trying to find a way to measure the distance between two rotation matrices. For my application, I need a measure that works in $SO(2)$ and one that works in $SO(3)$, but apart from that, I am much more curious about the general $SO(n)$ case. I can find measures for $SO(2)$ and $SO(3)$, but I have no idea how to approach the general case.
For $SO(3)$ I found this very insightful blog post. It uses $P$ and $Q$ to denote 3x3 rotation matrices and $*$ to denote the matrix transpose. It then measures distance using
$$\theta = \arccos\left(\frac{\mathrm{tr}(PQ^*) - 1}{2}\right).$$
For $SO(2)$ I can come up with a similar formula (with the help of Wikipedia):
$$\theta = \arccos\left(\frac{\mathrm{tr}(PQ^*)}{2}\right)$$
This makes me think that there should be a formula for $SO(n)$, but how do I work that out?
For context:
My application is to do inverse kinematics on a kinematic chain. Which is buzzword bingo for "I have a parameterized sequence of affine transformations (typically rotations and translations) and I am trying to find a set of parameters such that the transformation of the total sequence results in a desired translation and orientation".
In my case, I am interested in solving the above for a desired orientation only. One way to do so is via optimization, but for that, I need a way of measuring the distance between the current orientation and the desired orientation.
Note that there's no intrinsic notion of "distance" between rotations. Clearly you're referring to some kind of distance function, but there are many different distance functions on $SO(n)$, even when we add some sensible requirements. Which distance function to use largely depends on what you want to do with it. For instance, one simple choice is the Euclidean distance $d(A,B)=\|A-B\|$, where $\|A\|^2=\operatorname{tr}(A^TA)=\sum_{i,j}A_{ij}^2$, which is useful in some ways (it is bi-invariant and easy to compute), but not so useful in others (it doesn't have geodesics).
That said, the "standard" choice of distance is the geodesic distance arising from a particular Riemannian metric; that metric can be written as $g(U,V)=\operatorname{tr}(U^TV)$, where we interpret tangent vectors in $TSO(n)$ as $n\times n$ matrices in the standard way.
It's possible to compute this geodesic distance using eigenvalues. One can show that the eigenvalues of a rotation matrix consists of $1$s and pairs of complex conjugates of unit magnitude: $$ (\lambda_1,\cdots,\lambda_n)=\left(1,\cdots,1,e^{i\theta_1},e^{-i\theta_1}\cdots e^{i\theta_k},e^{-i\theta_k}\right) $$ Where $\theta_1,\cdots,\theta_k\in(-\pi,\pi]$ represent the rotation angles along $k\le n/2$ orthogonal planes. The geodesic distance discussed above has the form $$ d(A,B)=\sqrt{\theta_1^2+\cdots+\theta_k^2} $$ Where $\theta_1,\cdots,\theta_k$ are the rotation angles corresponding to the eigenvalues of $A^TB$.
The formulas you've listed agree with this distance. Computing it in higher dimension may require computing the eigenvalues of $A^TB$ directly though, as in dimension $\ge 4$ there will be multiple rotation angles, which can't be extracted as easily.