Let $S = \{M \in M_3(\mathbb{Z})|M^T \Omega M = \Omega \}$ where $\Omega = \begin{bmatrix} 1 & 0 & 0\\ 0 & 1 & 0 \\ 0 & 0 & -1 \end{bmatrix}$.
Let $x,y,z \in \mathbb{Z}$ s.t. $x^2+y^2=z^2$. I have to show that the vector
$\begin{bmatrix} x'\\ y'\\ z' \end{bmatrix}=M\begin{bmatrix} x\\ y\\ z \end{bmatrix}$
satisfies the same equation for every $M \in S$.
If I did understand the exercise correctly, I'll have to show that for every $M \in S$:
$M^T\begin{bmatrix} x'\\ y'\\ z' \end{bmatrix}M=\begin{bmatrix} x'\\ y'\\ z' \end{bmatrix}$
Did I have to switch the $\Omega$ with my new vector? Is that right? I think I somehow misunderstood the exercise.
no, the exercise that you typed just means this: For a column vector $v$ with entries $x,y,z,$ the quadratic form $x^2 + y^2 - z^2$ is just given by $$ v^T \Omega v $$ as this is a 1 by 1 matrix, that is, a number. All the exercise mentions is this: since $\Omega = M^T \Omega M,$ we find $$ v^T \Omega v = v^T M^T \Omega M v = (Mv)^T \Omega (Mv) = v'^T \Omega v' \; , \;$$ where i mean $v'$ to be the column vector $v' = Mv,$ with elements $x',y',z'.$
Meanwhile, from Magnus, given integers with $\alpha \delta - \beta \gamma = 1$ and $\alpha + \beta + \gamma + \delta$ even, we get an automorphism matrix for the diagonal matrix with elements ordered $-1,1,1$ as
$$ \left( \begin{array}{ccc} \frac{1}{2} \left( \alpha^2 + \beta^2 + \gamma^2 + \delta^2 \right) & \alpha \beta + \gamma \delta & \frac{1}{2} \left( \alpha^2 - \beta^2 + \gamma^2 - \delta^2 \right) \\ \alpha \gamma + \beta \delta &\alpha \delta + \beta \gamma &\alpha \gamma - \beta \delta \\ \frac{1}{2} \left( \alpha^2 + \beta^2 - \gamma^2 - \delta^2 \right) & \alpha \beta - \gamma \delta & \frac{1}{2} \left( \alpha^2 - \beta^2 - \gamma^2 + \delta^2 \right) \\ \end{array} \right) $$ Next, we need to interchange first and third columns and first and third rows.
$$ M = \left( \begin{array}{ccc} \frac{1}{2} \left( \alpha^2 - \beta^2 - \gamma^2 + \delta^2 \right) & \alpha \beta - \gamma \delta & \frac{1}{2} \left( \alpha^2 + \beta^2 - \gamma^2 - \delta^2 \right) \\ \alpha \gamma - \beta \delta &\alpha \delta + \beta \gamma &\alpha \gamma + \beta \delta \\ \frac{1}{2} \left( \alpha^2 - \beta^2 + \gamma^2 - \delta^2 \right) & \alpha \beta + \gamma \delta & \frac{1}{2} \left( \alpha^2 + \beta^2 + \gamma^2 + \delta^2 \right) \\ \end{array} \right) $$ This gives all such $M$ with positive determinant.
For example, $\alpha = 9, \beta = 4, \gamma = 2, \delta = 1,$ I get $$ M = \left( \begin{array}{ccc} 31 & 34 & 46 \\ 14 &17&22 \\ 34& 38 & 51 \\ \end{array} \right) $$
If I then take column vector $v$ with elements $(3,4,5)$ which is a Pythagorean triple, I get a column vector with elements $459,220,509$ This should be a Pythagorean triple as well. $459^2 + 220^2 = 259081.$ Then $509^2 = 259081.$ Hooray!