I have an orthogonal matrix $R$ and I am trying to differentiate the term $R^{\top}R$ wrt $R$. However, I am stuck at this point because the matrices are not symmetric. Anybody has an idea or trick. I also know that $\text{det}\left(R\right) = \text{det}\left(R^{\top}\right) = 1$. I have searched around and did not find anything.
\begin{array}{c} f = {R^ \top }R\\ \partial f = \left( {\partial {R^ \top }} \right)R + {R^ \top }\partial R\\ {\rm{vec}}\left( {\partial f} \right) = {\rm{vec}}\left( {\left( {\partial {R^ \top }} \right)R} \right) + {\rm{vec}}\left( {{R^ \top }\partial R} \right)\\ = {\rm{vec}}\left( {{R^ \top } \otimes {I_m}} \right){\rm{vec}}\left( {\partial {R^ \top }} \right) + {\rm{vec}}\left( {{I_m} \otimes {R^ \top }} \right){\rm{vec}}\left( {\partial R} \right) \end{array}
$R^T R =I$ for any orthogonal $R$ so the derivative within the set of orthogonal matrices is zero.
If we want the derivative within all matrices, then consider $$ (R+E)^T (R+E) = R^T R + E^T R + R^T E + EE^T = I + E^T R + R^T E +{\cal O}(||E||^2) $$ So the derivative is the map $$ E \mapsto E^T R + R^T E. $$