Do every two orthogonal matrices in $\text{SO}(n)$ lie in the same coset of $\text{SO}(2)$?

Question

Let $A,B \in \text{SO}(n)$.

Does there exist a homomorphism of Lie groups $\phi:\text{SO}(2) \to \text{SO}(n)$, such that $A,B$ lie in the same coset of $\phi(\text{SO}(2))\le \text{SO}(n)$?

Answer 1

The answer is yes if $n\leq 3$ and is no for $n\geq 4$.

As freakish says, you question is equivalent to the following:

Given any $p\in SO(n)$, is there a circle subgroup of $SO(n)$ containing $p$?

We will actually address this question for all compact Lie groups $G$ simultaneously.

Proposition 1: Suppose $G$ is a compact Lie group of rank $1$. Then any $p\in G$ lies on an $SO(2)\subseteq G$.

Proof: Recall that in any compact Lie group $G$, any $p\in G$ lies on a maximal torus $T$. If the rank of $G$ is $1$, then $T = S^1 = SO(2)$. $\square$

And now the more fun direction:

Proposition: If every $p\in G$ lies in an $SO(2)$ subgroup, then $G$ has rank $1$.

Proof: Let $p \in G$ with the property that the closure of the subgroup generated by $p$, $\overline{\langle p\rangle}$ is a maximal torus $T$. Such a $p$ always exists: beginning with any maximal torus $T'$, let $p = (\theta_1,...,\theta_k)\in T'$ with the $\{\theta_i\}$ linearly independent over $2\pi \mathbb{Q}$.

By assumption, $p\in SO(2)$ for some $SO(2)\subseteq G$. Because $SO(2)$ is a subgroup, $\langle p\rangle\subseteq SO(2)$. Because $SO(2)$ is closed (because it is compact), $\overline{\langle p \rangle}\subseteq SO(2)$. By construction, $T = \overline{\langle p \rangle}$, so $T\subseteq SO(2)$, so a maximal torus lies in $SO(2)$. That is, $G$ has rank $1$. $\square$