I think this question should be pretty straightforward but through a combination of being self-taught in group theory and being awful at geometry, something is escaping me.
Suppose I have an $n$-dimensional vector, $\mathbf{x} = (x_1, x_2, \ldots, x_n)^T$. I don't know the vector itself but I know that $\lVert \mathbf{x} \rVert^2 = G^2$, and that $\mathbf{x}\cdot \hat{n} = 0$ where $\hat{n}$ is, in my particular case, just $\frac{1}{\sqrt{n}}(1,1,\ldots,1)^T$, and I'd like to find a group of transformations that preserve those two facts, i.e. I'd like to characterize all vectors that satisfy these two constraints by characterizing the group of transformations that preserves these constraints.
If I didn't have the hyperplane constraint, obviously any transformation in the orthogonal group $O(n)$ will work. Since this hyperplane passes through the origin, its intersection with the n-sphere of radius G should be an (n-1)-sphere, so it seems "obvious" to me that there should be some subgroup of $O(n)$ (that's basically $O(n-1)$) that works here. However, I don't know how to show this.
In particular, my attempts seem to suggest that there's only a trivial group that preserves both things, which can't possibly be right. My argument is basically: to preserve the magnitude of the vector, I need to select $A \in O(n)$. Now to preserve the hyperplane constraint, I need to restrict to $A$ such that $(A \mathbf{x})^T \hat{n} = \mathbf{x}^T \hat{n}$. This seems to imply that $\mathbf{x}^T(A^T - I_n)\hat{n} = 0$, which is only satisfied if $A^T = I_n$.
So where is my argument going wrong? What's the right way to think about this problem?
No. This is satisfied, e.g., when $(A^T-I)\hat{n}=0$, i.e., when $\hat{n}$ is an eigenvector of $A^T$ corresponding to the unit eigenvalue.
Anyway, you need $\|Ax\|=\|x\|$ and $Ax\perp\hat{n}$ whenever $x\perp\hat{n}$. This means $\{\hat{n}\}^\perp$ is an invariant subspace of $A$ and on which $A$ is an isometry. So, if you pick a fixed orthogonal matrix $U$ whose last column is $\hat{n}$, $A$ must be in the form of $$ A=U\pmatrix{Q&\ast\\ 0&\ast}U^T $$ where $Q\in O(n-1)$. Since you want the candidate $A$s to form a multiplicative group, either all members of the group are nonsingular or all members are singular. Thus the group is a subgroup of either $$ \left\{U\pmatrix{Q&v\\ 0&a}U^T:\ Q\in O(n-1),\,a\ne0\right\} $$ or $$ \left\{U\pmatrix{Q&0\\ 0&0}U^T:\ Q\in O(n-1)\right\}. $$