By classical results in invariant theory, we know the invariant ring $\mathbb{C}[X_1, X_2, X_3]^{\mathrm{SO}_3(\mathbb{C})}$ is equal to $\mathbb{C}[X_1^2 + X_2^2 + X_3^2]$. Here $\mathbb{C}X_1 \oplus \mathbb{C}X_2 \oplus \mathbb{C}X_3$ is the fundamental representation of $\mathrm{SO}_3(\mathbb{C})$.
My question is, what about $\mathbb{Z}[X_1, X_2, X_3]^{\mathrm{SO}_3(\mathbb{Z})}$?
More generally, view $\mathrm{SO}_3$ as an affine group scheme, can we describe the functor $$R\longmapsto R[X_1, X_2, X_3]^{\mathrm{SO}_3(R)}$$ from the category of commutative rings to itself?
$SO_3(\mathbb{Z})$ is generated by
$\left[ \begin{array}{rrr} -1 & 0 & 0 \\ 0 & -1 & 0 \\ 0 & 0 & 1 \end{array} \right],\left[ \begin{array}{rrr} 0 & 1 & 0 \\ -1 & 0 & 0 \\ 0 & 0 & 1 \end{array} \right],\left[ \begin{array}{rrr} 0 & 0 & 1 \\ 1 & 0 & 0 \\ 0 & 1 & 0 \end{array} \right]$.
You can conclude that the invariant ring is $\mathbb{Z}[\,x_1^2 x_2^2 x_3^2\,,\, x_1^2+x_2^2+x_3^2\,,\,x_1^2 x_2^2+x_2^2 x_3^2+x_3^2 x_1^2\,,\,x_1x_2x_3(x_1^2-x_2^2)(x_2^2-x_3^2)(x_3^2-x_1^2)\,]$.
$\\$
How about an arbitrary ring $R$ in general? A fascinating question! If you're interested, maybe consider finite fields? I haven't thought much about this, but good luck!