Let $R$ be a $3\times 3$ orthogonal matrix. Let $v$ be the unit vector such that $Rv=v$ (upto sign change). Consider any unit vector $u$ such that $u^{T}v=0$ where $T$ stands for transpose. Show that $u^{T}Ru = \frac12(trace(R)-1)$.
The problem arises in finding the angle of rotation given a rotation-transformation matrix. The vector $v$ would be the axis of rotation since it is invariant under the transformation $R$. The angle of rotation can be obtained by observing how much $u$ rotated by. That is, $\cos(\theta)=u^TRu$ where $\theta$ is the rotation angle. I'm not able to understand how the trace of $R$ enters the formula.
Analyze Rodrigues formula for rotation.
Let it be in a such form: $R(v,\theta)=I+\sin(\theta)S(v)+(1-\cos(\theta))S^2(v)$.
The expression $\sin(\theta)S(v)$ is the skew-symmetrical part of rotation $R$,
$I+(1-\cos(\theta))S^2(v)$ is its symmetrical part.
Trace of any skew-symmetrical matrix is equal $0$ so $\text{trace}(R)=\text{trace}(\text{sym(R)})$.
Additionally $S^2(v)=vv^T-I$, where $\Vert v \Vert=1$, - notice that in this case we have also $\text{trace}(vv^T)=1$.
Now calculating $\text{trace}(R)$ we obtain
$\text{trace}(R) =\text{trace} (I)+\text{trace}((1-\cos(\theta))(vv^T-I))=3-2(1-\cos(\theta))$.
Hence $\text{trace}(R)=1+2 \cos(\theta) $
It can be also other way of proving it.
Rotation matrix $R$ could be transformed to the other basis where axis of rotation would be z-axis.
Such operation doesn't affect the trace
(algebraically it is expressed as $R(v,\theta)=R_{Trans}R(z,\theta)R_{Trans}^{-1}$ ).
For rotation about z-axis we have $R(z,\theta)=\begin{bmatrix}\cos(\theta) & -\sin (\theta) & 0 \\ \sin (\theta) & \cos (\theta) & 0 \\ 0 & 0 & 1 \end{bmatrix}$.
From it $\text{trace}(R)=1+2 \cos(\theta) $ follows.