The automorphisms of $\mathbb R$ form a group $\mathbb Q^* \times \mathrm {GL}_{\mathfrak c}(\mathbb Q)$, where $\mathrm {GL}_{\mathfrak c}(\mathbb Q)$ is the $\mathfrak c$-dimensional general linear group over $\mathbb Q$, assuming existence of a Hamel basis.
I have worked out the automorphism group of $\mathbb Q/\mathbb Z$, which should be $\prod_{p} \mathbb Z_p^*$, where $\mathbb Z_p$ is the $p$-adic integers, and $\mathbb Z^*_p$ is the $p$-adic integers that has absolute value $1$.
However, I have difficulty seeing how we can characterize the automorphism on irrational elements of $\mathbb R/\mathbb Z$: even if we somehow use the axiom of Choice to choose a sort of "maximally independent set", how can we be sure that the resulting homomorphism will be an automorphism? In the case of $\mathrm{Aut}(\mathbb R)$, the general linear group takes care of this. I know little about modules, and I'm struggling to create an analogous automorphism group in the case of $\mathbb Q/\mathbb Z$.
Is there any existing material on this? By the way, is there a characterization of the automorphism of $\mathrm{SO}(n)$ as a group?
You can write $\mathbb{R}/\mathbb{Z}=\mathbb{Q}/\mathbb{Z}\oplus G$, where $G$ is a torsion-free divisible group, hence isomorphic to a direct sum of copies of $\mathbb{Q}$. From cardinality considerations, we see that $G\cong\mathbb{R}$.
Set $T=\mathbb{Q}/\mathbb{Z}$, for simplicity. Then we can describe endomorphisms of $T\oplus G$ as matrices of the form $$ \begin{bmatrix} \alpha & \beta \\ 0 & \gamma \end{bmatrix} \quad\text{where} \quad \left\{\begin{aligned} \alpha &\!: T \to T \\ \beta &\!: G\to T \\ \gamma &\!: G\to G \end{aligned}\right. $$ and the composition is the formal matrix product. So, in order to have an automorphism it is necessary and sufficient that $\alpha$ and $\gamma$ are automorphisms, because in this case we have $$ \begin{bmatrix} \alpha & \beta \\ 0 & \gamma \end{bmatrix}^{-1}= \begin{bmatrix} \alpha^{-1} & -\alpha^{-1}\beta\gamma^{-1} \\ 0 & \gamma^{-1} \end{bmatrix} $$