I am considering the group of hyperbolic rotation matrices $G=\{A\in M_{3\times 3}(\mathbb{R}): A^TDA=D \}$, where $D=\begin{bmatrix} 1&0&0\\ 0&1&0\\ 0&0&-1\\ \end{bmatrix}$. It is generated by the following three types of matrices, namely, the rotations about $x$-axis, $y$-axis and $z$-axis for the hyperboloid $\mathbb{H}^2:=\left\{\begin{bmatrix} x\\ y\\ z\\ \end{bmatrix}: x^2+y^2-z^2=-1, z>0 \right\}$:
$X(\theta) :=\begin{bmatrix} 1&0&0\\ 0&\cosh\theta&\sinh\theta\\ 0&\sinh\theta&\cosh\theta\\ \end{bmatrix}$, $Y(\theta) :=\begin{bmatrix} \cosh\theta&0&\sinh\theta\\ 0&1&0\\ \sinh\theta&0&\cosh\theta\\ \end{bmatrix}$ and $Z(\theta) :=\begin{bmatrix} \cos\theta&-\sin\theta&0\\ \sin\theta&\cos\theta&0\\ 0&0&1\\ \end{bmatrix}$.
See p.7-8 of this note for more details.
I am trying to compare this with $SO(3)$.
It is well-known that $A\in SO(3)\iff A^TA=I \iff \langle A x,A y\rangle_{\mathbb E^3}=\langle x,y\rangle_{\mathbb E^3}$, where $\langle x,y\rangle_{\mathbb E^3}=x^Ty$ and $\mathbb{E}^3$ is Euclidean 3-space.
It is also known that $A\in G\iff A^TDA=D \iff \langle Ax,Ay\rangle_{\mathbb M^3}=\langle x,y\rangle_{\mathbb M^3}$, where $\langle x,y\rangle_{\mathbb M^3}=x^TDy$ and $\mathbb{M}^3$ is Minkowski 3-space.
For $A=\begin{bmatrix} a_{11}&a_{12}&a_{13}\\ a_{21}&1&a_{23}\\ a_{31}&a_{32}&a_{33}\\ \end{bmatrix}\in SO(3)$, we have $\langle\begin{bmatrix} a_{12}\\ 1\\ a_{32}\\ \end{bmatrix},\begin{bmatrix} a_{12}\\ 1\\ a_{32}\\ \end{bmatrix}\rangle_{\mathbb{E}^3} =\langle\begin{bmatrix} 0\\ 1\\ 0\\ \end{bmatrix},\begin{bmatrix} 0\\ 1\\ 0\\ \end{bmatrix}\rangle_{\mathbb{E}^3}$. Then $a_{12}^2+1+a_{32}^2=1$ and hence $a_{12}=a_{32}=0$. Since we also have $A^T\in SO(3)$, we can get $a_{21}=a_{23}=0$ by the same reason. Further argument can show $A=\begin{bmatrix} \cos\theta&0&\sin\theta\\ 0&1&0\\ -\sin\theta&0&\cos\theta\\ \end{bmatrix}$ for some $\theta$, which means $A$ is a rotation about $y$-axis.
Question 1: Is it the same for $A\in G$? Namely, if $A\in G$ and $a_{22}=1$, can we show $A=\begin{bmatrix} \cosh\theta&0&\sinh\theta\\ 0&1&0\\ \sinh\theta&0&\cosh\theta\\ \end{bmatrix}$ for some $\theta$? It seems not that trivial to me since I get $a_{12}^2+1-a_{32}^2=1$ instead.
Question 2: If not, any counterexample?
The following matrix is an counterexample: \begin{bmatrix} \tfrac{1}{\sqrt{2}}& 1&-\tfrac{1}{\sqrt{2}}\\ -\tfrac{1}{\sqrt{2}}& 1&-\tfrac{1}{\sqrt{2}}\\ 0&-1&\sqrt{2}\\ \end{bmatrix}