Let $\vec x$ be a non-zero vector in $\mathbb R^3$ and let $G=\{A \in GL(3, \mathbb R) \mid A\vec x = \vec x\}$. Suppose that $\vec x = [1\ 0\ 0]^T$. Describe all the matrices in $G$, as defined above, that are also orthogonal matrices.
This is my attempt: $$ \begin{pmatrix} a & b & c \\ d & e & f \\ g & h & i \\ \end{pmatrix}\begin{pmatrix} 1\\ 0\\ 0\\ \end{pmatrix}=\begin{pmatrix} a\\ d\\ g\\ \end{pmatrix}=\begin{pmatrix} 1\\ 0\\ 0\\ \end{pmatrix} $$ Hence $a=1, d=0, g=0$
Then we need to find values or restrictions of $b, c, e, f, h, i$ such that $$ \begin{pmatrix} 1 & b & c \\ 0 & e & f \\ 0 & h & i \\ \end{pmatrix}\begin{pmatrix} 1 & 0 & 0 \\ b & e & h \\ c & f & i \\ \end{pmatrix}=\begin{pmatrix} 1 & 0 & 0 \\ b & e & h \\ c & f & i \\ \end{pmatrix}\begin{pmatrix} 1 & b & c \\ 0 & e & f \\ 0 & h & i \\ \end{pmatrix}=I_3 $$ Here $$ \begin{pmatrix} 1 & b & c \\ 0 & e & f \\ 0 & h & i \\ \end{pmatrix}\begin{pmatrix} 1 & 0 & 0 \\ b & e & h \\ c & f & i \\ \end{pmatrix}= \begin{pmatrix} b^2+c^2+1 & cf+be & ic+bh \\ cf+be & f^2+e^2 & if+he \\ ic+bh & if+he & h^2-1 \\ \end{pmatrix} $$ Where we can determine $h^2-1=1, h^2=2$
Then$$ \begin{pmatrix} 1 & 0 & 0 \\ b & e & h \\ c & f & i \\ \end{pmatrix}\begin{pmatrix} 1 & b & c \\ 0 & e & f \\ 0 & h & i \\ \end{pmatrix}= \begin{pmatrix} 1 & b & c \\ b & b^2+h^2+e^2 & bc+ih+fe \\ c & bc+ih+fe & c^2+f^2-1 \\ \end{pmatrix} $$ Where we find that $b=c=0, b^2+h^2+e^2=1$, but then $e^2$ has to be $-1$, which contradicts to such matrix should be $\in GL(3, \mathbb R)$
Finally conclude that such matrix does not exist. Am I correct?
Since $A^TA=I$ we have $A^Tx = x$ so the subspace spanned by $x$ is $A$ and $A^T$ invariant, so $A$ has the form $\begin{bmatrix} 1 & 0 & 0 \\ 0 & a & b \\ 0 & c & d \end{bmatrix}$.
Since $A$ is orthogonal, we see that the submatrix $\begin{bmatrix} a & b \\ c & d \end{bmatrix}$ is must be orthogonal and since it is $\mathbb{R}^2 \to \mathbb{R}^2$ we see that it must be a rotation (proper or improper).