Let $S = \{M \in M_3(\mathbb{Z})\mid M^T \Omega M = \Omega \}$, where
$$\Omega = \begin{bmatrix} 1 & 0 & 0\\ 0 & 1 & 0 \\ 0 & 0 & -1 \end{bmatrix}$$
I'm trying to prove that $S$ is a group under matrix-multiplication.
Proof:
- Show it is closed under multiplication
Let $M,N \in S$. This means $M^T \Omega M = \Omega$ and $N^T \Omega N = \Omega$. I have to show that $(MN)^T \Omega (MN) = \Omega$. This would be equivalent to $N^TM^T \Omega MN = \Omega$ which is the same as $N^T \Omega N = \Omega$.
So it is closed under matrix multiplication.
Now, I struggle to show that there exists an inverse. I will have to prove that all of these matrices in $S$ are invertible and that their inverse is in $S$ as well.
Can you give me an idea on how to do this?
If $M^{\top}\Omega M=\Omega$ then $M$ is invertible in $M_3(\Bbb{Z})$ because $$\det(M^{\top}\Omega M)=\det(\Omega)=-1,$$ and hence $\det(M)=\pm1$. It follows that $$M^{-\top}\Omega M^{-1}=M^{-\top}(M^{\top}\Omega M)M^{-1}=(M^{-\top}M^{\top})\Omega(MM^{-1})=\Omega,$$ so $M^{-1}\in S$. You may also want to mention that $I\in S$, even though this follows from the above.