I need to compute the determinant of the jacobian matrix of the function $f: SO(n)\times SO(n) \rightarrow SO(n)\times SO(n)$ given by $f(P,V) = (PV, P^TV)$. I've found the jacobian if we extend $f$ to the whole of $M(n)$, but I'm not sure how to find the jacobian of the restricted map.
Any help is greatly appreciated!
In each small neighbourhood of $(P,V)\in SO(n,\mathbb{R})\times SO(n,\mathbb{R})$, let us represent each point in the neighbourhood by $(e^XP, e^YV)$ for some skew symmetric matrices $X$ and $Y$. Then $(e^XP, e^YV)-(P,V)=(XP,YV)+o(\sqrt{\|XP\|_F^2+\|YV\|_F^2})$ (where $\|\cdot\|_F$ denotes the Frobenius norm) and \begin{align*} &\left(e^XPe^YV,\ (e^XP)^T e^YV\right) - \left(PV,\ P^T V\right)\\ =&\left( (e^X Pe^YP^T)PV,\ \left(P^Te^{-X} e^YP\right)P^TV\right) - \left(PV,\ P^T V\right)\\ =&\left((X + PYP^T)PV,\ \left(-P^TXP+P^TYP\right)P^TV\right)+o(\sqrt{\|XP\|_F^2+\|YV\|_F^2}). \end{align*} Therefore, the derivative of $f$ at $(P,V)$ is the linear operator $Df(P,V):\mathfrak{so}(n,\mathbb{R})P\times \mathfrak{so}(n,\mathbb{R})V\to\mathfrak{so}(n,\mathbb{R})PV\times \mathfrak{so}(n,\mathbb{R})P^TV$ defined by $$ (XP,YP)\mapsto\left((X + PYP^T)PV,\ \left(-P^TXP+P^TYP\right)P^TV\right). $$ That is, $Df(P,V)=h\circ g\circ s$ where \begin{align*} s(XP,YP)&=(X,Y),\\ g(X,Y)=(Z,W)&=\left(X + PYP^T,\ -P^TXP+P^TYP\right),\\ h(Z,W)&=\left(ZPV,\ WP^TV\right) \end{align*} and $\det Df(P,V)=\det(h)\det(g)\det(s)$. Using Kronecker product, we can express $s$ and $g$ as \begin{align*} g\left(\mathrm{vec}(X,Y)\right) = \pmatrix{I\otimes I& P\otimes P\\ -P^T\otimes P^T&P^T\otimes P^T}\mathrm{vec}(X,Y). \end{align*} Therefore $\det(g)=\det\left((I\otimes I)(P^T\otimes P^T) - (P\otimes P)(P^T\otimes P^T)\right)=\det(P\otimes P + I_{n^2})$. Similarly, since $\det(P)=\det(V)=1$, we have $\det(h)=\det(s)=1$. Hence $$\det Df(P,V)=\det(P\otimes P+I_{n^2}).$$
Of course, you may also express each point in a small neighbourhood by $(Pe^X,Ve^Y)$. In this case, we obtain \begin{align*} &\left(Pe^X Ve^Y,\ (Pe^X)^T Ve^Y\right) - \left(PV,\ P^T V\right)\\ =&\left(PV (V^Te^X Ve^Y),\ P^TV \left((P^T V)^T e^{-X} (P^T V)e^Y\right)\right) - \left(PV,\ P^T V\right)\\ =&\left(PV (V^T X V + Y),\ P^TV \left(-(P^T V)^T X (P^T V) + Y\right)\right)+o(\sqrt{\|X\|_F^2+\|Y\|_F^2}). \end{align*} and hence $Df(P,V)$ is given by $$ (PX,VY)\mapsto\left(PV (V^T X V + Y),\ P^TV \left(-(P^T V)^T X (P^T V) + Y\right)\right). $$ Again, $\det Df(P,V)=\det(P\otimes P+I_{n^2})$.