My prime problem is to minimize a cost function $$f(X) = \left(v - Xu\right)^T\left(B + XAX^T\right)^{-1}\left(v - Xu\right)$$ where $v,u\in \mathbb{R}^3$,$A,B\in \mathbb{R}^{3\times3}$ are positive definite, and the variable $X \in SO(3)$ is a $3\times3$ rotation matrix satisfying $X^TX=I$.
The dual problem is finding the Jacobian matrix of $f(X)$. I would like to get an explicit expression, but I found no proper way to achieve that limited by my knowledge.
I will be very appreciated if someone could show me how to make this derivation step by step. Thanks a lot!
I tried http://www.matrixcalculus.org and it gave the very complicated answer
$ \dfrac{\partial f}{\partial X} = -(\mathrm{inv}(B+X\cdot A\cdot X^\top )\cdot (v-X\cdot u)\cdot u^\top +\mathrm{inv}(B^\top +X\cdot A^\top \cdot X^\top )\cdot (v-X\cdot u)\cdot ((v-X\cdot u)^\top \cdot \mathrm{inv}(B^\top +X\cdot A^\top \cdot X^\top )\cdot X\cdot A^\top )+\mathrm{inv}(B+X\cdot A\cdot X^\top )\cdot (v-X\cdot u)\cdot ((v-X\cdot u)^\top \cdot \mathrm{inv}(B+X\cdot A\cdot X^\top )\cdot X\cdot A)+\mathrm{inv}(B^\top +X\cdot A^\top \cdot X^\top )\cdot (v-X\cdot u)\cdot u^\top )$
without the steps.
Where the input is (v-X*u)'*inv(B+X*A*X')*(v-X*u), where the prime denotes the transpose.
But it still doesn't give an explicit expression for $X$, when the derivative equals the $0$ matrix.
Maybe https://www.geno-project.org will help.