Let $\mathbf{v}:=(a,b,c,d)^T\in\mathbb{R}^4.$ Then does the set of points in $\mathbb{R}^4$ $$S:=\{a,b,c,d\;|\;ab>cd \;\wedge\; a+b<0\}$$ have any non-trivial rotational symmetries about the origin, where non-trivial means that the rotations are not of order $1$ (identity) or $2$ (reflections)? I.e., does there exist any orthogonal $4\times 4$ matrices $\mathbf{M}$ such that if $\mathbf{v}\in S$ then $\mathbf{Mv}\in S$?
I suspect that there does not, but how can I approach this? Any ideas and/or observations are welcome.
Thanks.
Edit: Note that $\mathbf{v}$ can be rewritten as a matrix $\mathbf{V}:=\pmatrix{a & c \\ d & b}$ such that $S$ is defined as $$S:=\{\mathbf{A}\;|\;\det\mathbf{A}>0\;\wedge \; \mathrm{tr}\,\mathbf{A}<0\},$$ but then the rotation matrix becomes more obscure.
If $M$ maps $S$ into $S$, it must map its closure $\overline{S}=\{(a,b,c,d): ab\ge cd,\ a+b\le0\}$ into $\overline{S}$. Note that a one-dimensional linear subspace is contained in $\overline{S}$ if and only if it is the linear span of a vector in one of the two sets below: \begin{align*} X&=\{(0,0,c,d):\ (c,d)\ne(0,0),\ cd\le0\},\\ Y&=\{(a,-a,c,d):\ a\ne0,\, -a^2\ge cd\}. \end{align*} Since $M$ preserves one-dimensional linear subspaces in $\overline{S}$, it must map $X\cup Y$ into $X\cup Y$.
Vectors in $X$ and $Y$ differ in a crucial way. It is not hard to show that:
(a) if $x\in X$ is normal to another vector $x'\in X\cup Y$, then $x'$ must also belong to $X$ and both $x,x'$ are scalar multiples of $\pm e_3$ or $\pm e_4$;
(b) if $y\in Y$ is normal to another vector $y'\in X\cup Y$, then $y$ must be a scalar multiple of $(1,-1,c,-\frac 1c)$ for some $c\ne0$ and $y'$ must be a scalar multiple of $(1,-1,-\frac 1c,c)$.
Therefore, every pair of orthonormal vectors $(y,y')\in Y^2$ is a limit point in $(X\cup Y)^2$, while every pair of orthonormal vectors $(x,x')\in X^2$ is isolated. Since $M$ preserves pairs of orthonormal vectors as well as converging sequences, it must preserve pairs of orthonormal vectors in $X$, and also pairs of orthonormal vectors in $Y$.
Hence (a) implies that the unordered set $\{Me_3,Me_4\}$ is equal to $\pm\{e_3,\pm e_4\}$. As $M(e_3-e_4)$ must reside in $\overline{S}$, its value cannot be $\pm(e_3+e_4)$. Hence $(Me_3,Me_4)=\pm(e_3,e_4)$ or $\pm(e_4,e_3)$ and in turn $M$ must map $X$ onto $X$.
By (b), there are only two pairs of mutually orthogonal vectors $y,y'\in Y$ that satisfy $\|y\|=\|y'\|=2$ and $\langle y,e_3\rangle=-\langle y,e_4\rangle=1$, namely, $y=(a,-a,1,-1)$ and $y'=(a,-a,-1,1)$ where $a=\pm1$. Since $(Me_3,Me_4)=\pm(e_3,e_4)$ and $M$ preserves norms, inner products and mutually orthogonal vectors of $Y$, we see that $My=\pm(1,-1,0,0)+M(e_3-e_4)$ and hence $M(e_1-e_2)=\pm(e_1-e_2)$.
But then we get $M(e_1+e_2)=\pm(e_1+e_2)$, because $M$ is an isometry and $e_1+e_2$ is orthogonal to $\operatorname{span}\{e_1-e_2,e_3,e_4\}$, which is an invariant subspace of $M$.
Using the fact that $M$ is an isometry, $M(e_1-e_2)=\pm(e_1-e_2)$ and $M(e_1+e_2)=\pm(e_1+e_2)$, one can easily infer that $(Me_1,Me_2)=(e_1,e_2),(-e_1,-e_2),(e_2,e_1)$ or $(-e_2,-e_1)$. As $M$ also maps $S$ into $S$ (and preserves quadruples $(a,b,c,d)$ such that $a+b<0$), we conclude that $(Me_1,Me_2)$ can only be $(e_1,e_2)$ or $(e_2,e_1)$.
Consequently, $M=\pmatrix{M_1&0\\ 0&M_2}$, where $M_1\in\{I,R\},\ M_2\in\{I,-I,R,-R\}$ and $R=\pmatrix{0&1\\ 1&0}$. In other words, the answer to your question is negative.