I'm solving a problem from the book, Mathematics for 3D Game Programming and Computer Graphics, Third Edition, by Eric Lengley. The problem goes:
Let $N$ be the normal vector to a surface at a point $P$, and let $S$ and $T$ be tangent vectors at the point $P$ such that $S \times T = N$. Given an invertible 3 $\times$ 3 matrix $M$, show that $(MS) \times (MT) = (\text{det}M(M^{-1})^{T}(S \times T)$, supporting the fact that normals are correctly transformed by the inverse transpose of the matrix $M$.
The author provided a hint stating we can represent $(MS) \times (MT)$ as
$$ (MS) \times (MT) = \begin{bmatrix} 0 & -(MS)_{z} & (MS)_{y} \\ (MS)_{z} & 0 & -(MS)_{x} \\ -(MS)_{y} & (MS)_{x} & 0 \end{bmatrix} MT $$
We then find a matrix $G$ such that
$$ GU = \begin{bmatrix} 0 & -(MS)_{z} & (MS)_{y} \\ (MS)_{z} & 0 & -(MS)_{x} \\ -(MS)_{y} & (MS)_{x} & 0 \end{bmatrix} M $$
where
$$ U = \begin{bmatrix} 0 & -S_{z} & S_{y} \\ S_{z} & 0 & -S_{x} \\ -S_{y} & S_{x} & 0 \end{bmatrix} $$
and show that $G = (\text{det}M)(M^{-1})^{T}$ to solve the problem.
I am aware that there is an alternative solution to this problem, but I would like to solve it through the hints provided. Unfortunately, I am only able to go as far doing:
$$ G = \begin{bmatrix} 0 & -(MS)_{z} & (MS)_{y} \\ (MS)_{z} & 0 & -(MS)_{x} \\ -(MS)_{y} & (MS)_{x} & 0 \end{bmatrix} M U^{-1} $$
At this point, I do not know how to proceed with showing that $G = (\text{det}M)(M^{-1})^{T}$. How would you proceed? I'd like to ask for hints on solving the problem.
Using the scalar-triple product: we know that $$ A \cdot (B \times C) = \det[A \ \ B\ \ C], $$ where $A \cdot B = A^TB$ denotes a dot-product (and ${}^T$ denotes a transpose). With that, we can deduce the entries of $(MS) \times (MT)$.
Let $e_1,e_2,e_3$ denote the $x,y,z$ unit vectors. Using the above and the rule $\det(PQ) = \det(P)\det(Q)$, we see that for all $i$,
$$ \begin{align} (Me_i) \cdot [(MS) \times (MT)] & = \det[Me_i \ \ MS\ \ MT] \\ & = \det(M) \det[e_i\ \ S\ \ T] \\ & = \det(M) (e_i \cdot (S\times T)) \\ & = \det(M) (S\times T)_i. \end{align} $$ That is, we have $$ \begin{align} M^T ((MS) \times (MT)) & = \pmatrix{(Me_1)^T\\ (Me_2)^T\\ (Me_3)^T}((MS) \times (MT)) \\ & = \det(M)\pmatrix{(S \times T)_1\\ (S \times T)_2 \\ (S \times T)_3} \\ & = \det(M) (S \times T). \end{align} $$ Solving for $S \times T$, we have $$ M^T ((MS) \times (MT)) = \det(M) (S \times T) \implies (MS) \times (MT) = \det(M)M^{-T} (S \times T), $$ which was what we wanted.