Given the field $F$, $n \in \mathbb{N}$ and $U_{Q} := \{A \in F^{nxn}: A^TQA = Q \}$ for $Q \in F^{nxn} $
I want to show that:
For all $$M, S \in GL(n,K) \ and \ N := S^TMS$$ the following function is well-defined and a group homomorphism between the subgroups $U_{M}$ and $U_{N}$ of $GL(n,K)$:
$$ \phi:U_{M} \rightarrow U_{N}, \phi(A) = S^{-1}AS$$
Per definition of the homomorphism, I will have to show the preservation of the operation between $U_{M}$ and $U_{N}$. But I struggle to see what the exact operation here is and why it is important to write $\phi(A) = S^{-1}AS$ instead of $\phi(A) = A$.
As for the well-definedness part: I understand, that the goal is to prove that the function is unambiguous, but again I struggle to find a starting point here.
The operation is matrix multiplication: you need to prove that $\phi(AB)=\phi(A)\phi(B)$, that is $S^{-1}ABS=(S^{-1}AS)(S^{-1}BS)$.
(NB $S^{-1}AS\ne S$ in general, as multiplication is non-commutative).
For well-definedness, you need to prove that $\phi(A)\in U_N$ when $A\in U_M$, that is $ \phi(A)^TN\phi(A)=N$ whenever $A^TMA=M$. It may help to note that $$\phi(A)^TNA=(S^TA^T(S^T)^{-1})(S^TMS)(S^{-1}AS).$$