Define a density function $\rho(\boldsymbol{x}) : \mathbb{R}^3\mapsto \mathbb{R}$, now I want to compute the derivative of $\rho(\boldsymbol{Rx})$ with respect to $\boldsymbol{R}\in SO(3)$.
I have been trying to apply the identities in "Micro Lie theory" [1], but I come up with nonsense. For example, it is stated that $J_X^Y$ is the Jacobian of $Y$ w.r.t $X$, then $J^Z_X=J^Z_Y J_X^Y$. That seems logical until you consider that Y, X and Z can have different domains.
In the present case, this chain rule would give us $$J^{\rho(\boldsymbol{Rx})}_\boldsymbol{R}=J^{\rho(\boldsymbol{Rx})}_\boldsymbol{Rx} \,\cdot J^\boldsymbol{Rx}_\boldsymbol{R}$$ where the first Jacobian is in $\mathbb{R}^{1\times{}3}$ and the second is in $\mathbb{R}^{3\times{}3}$; this would mean that the derivative of ${\rho(\boldsymbol{Rx})}$ w.r.t $\boldsymbol{R}$ is a vector in $\mathbb{R}^3$, which makes little sense because the derivative should be composable with a rotation.
What am I misunderstanding?
[1] Sola et al, A micro Lie theory for state estimation in robotics https://arxiv.org/abs/1812.01537
One may identify $\mathfrak{so}(3)$ with $\mathbb{R}^3$ by the map $$\phi:\begin{bmatrix}0&-z&y\\ z & 0 &-x\\ -y & x&0\end{bmatrix}\mapsto \begin{bmatrix}x\\ y\\z\end{bmatrix}$$ We If we take one of the coordinate rotations $R=\begin{bmatrix} \cos\theta & -\sin\theta &0\\ \sin\theta &\cos\theta &0\\0&0&1\end{bmatrix}$ we see that $$R\begin{bmatrix}0&-z&y\\ z & 0 &-x\\ -y & x&0\end{bmatrix}R^T=\begin{bmatrix}0 &-z &y\cos\theta +x\sin\theta \\ z& 0 &y\sin\theta -x\cos\theta \\ -y\cos\theta-x\sin\theta & -\sin\theta y+x\cos\theta & 0\end{bmatrix}$$ so $\phi(R\xi R^T)=R\cdot \phi(\xi)$ for this particular $R\in SO(3)$. It turns out that this is true generally in $SO(3)$, that $\phi(A\xi A^T)=A\cdot \phi(\xi)$ for all $\xi\in \mathfrak{so}(3)$ and $A\in SO(3)$. In representation theoretic terms this means that $\phi$ is not only an isomorphism, but an equivariant isomorphism with respect to the adjoint action of $SO(3)$ on $\mathfrak{so(3)}$ and the natural action on $\mathbb{R}^3$.
This means that we can view our function $\rho: \mathbb{R}^3\to \mathbb{R}$ as a function $\mathfrak{so}(3)\to \mathbb{R}$. In this way, we can write our map as $R\mapsto \rho(Rx R^T)=\rho(\mathrm{Ad}_{R}x)$ committing the abuse of notation by identifying an element of $\mathbb{R}^3$ with an element of $\mathfrak{so}(3)$. Recall that for all $R\in SO(3)$, $T_RSO(3)=R\cdot\mathfrak{so}(3)$ (this multiplication is matrix multiplication). Now to compute the derivative, we see that \begin{align}T_Rf_x(R\xi)&=\frac{d}{dt}\bigg\vert_{t=0}f_x(R\exp(t\xi))\\ &=\frac{d}{dt}\bigg\vert_{t=0}\rho(R\exp(t\xi) x\exp(-t\xi)R^T)\\ &=\rho'(RxR^T)\cdot\frac{d}{dt}\bigg\vert_{t=0}R\exp(t\xi)x\exp(-t\xi)R^T\\ &=\rho'(RxR^T)\cdot R\bigg(\frac{d}{dt}\bigg\vert_{t=0}\exp(t\xi)x\exp(-t\xi)\bigg)R^T\end{align} Now, generically $\frac{d}{dt}\bigg\vert_{t=0}\exp(t\xi)x\exp(-t\xi)=\frac{d}{dt}\bigg\vert_{t=0}\mathrm{Ad}_{\exp(t\xi)}x=[\xi,x]$ (This can be taken as the definition of the Lie bracket of a Lie algebra of a Lie group) Giving $$T_Rf_x(R\xi)=\rho'(RxR^T)R[\xi,x]R^T$$
One may show that our map $\phi$ satisfies another property, namely that it sends the Lie bracket to the cross product, $\phi([\xi,x])=\phi(\xi)\times \phi(x)$ Which allows us to write our equation for the derivative under our identification as $$T_Rf_x(R\xi)=\rho'(R\cdot x) R\cdot(\phi(\xi)\times x)$$