Let $G$ be a finite nontrivial subgroup of $\text{SO}(3)$. Let $X$ be the set of points on the unit sphere in $\mathbb{R}^3$ which are fixed by some nontrivial rotation in $G$. I have two questions.
- How do I see that $G$ acts on $X$ and that the number of orbits is either $2$ or $3$?
- What is $G$ is there are only $2$ orbits?
Any nontrivial rotation in $\text{SO}_3$ has a collection of fixed points being a $1$-dimensional subspace of $\mathbb{R}^3$ — its axis of rotation. Such a subspace meets the unit sphere in exactly two antipodal points, and thus $X$ consists of the collection of these antipodal points — which are often called poles. To show $G$ acts on $X$, if $P \in X$ is a pole of $g \in G$ and $h \in G$ is some other element, then $h(P)$ is fixed by $hgh^{-1} \in G$, and thus $h(P)$ is a pole of another element of $G$, and hence $h(P) \in X$.
Since every nontrivial element of $G$ fixes exactly two points of $X$, whilst the identity fixes all of $X$, we can count that the number of distinct orbits of the action is:
$$ \text{number of orbits} = \frac{1}{|G|} \sum_{g\in G} |\text{Fix}_g(X)| = \frac{1}{|G|} (2(|G| - 1) + |X|).$$
If we let the number of orbits be $m$ and let $x_1, \ldots , x_m$ be representatives for each orbit in $X$, then we can write:
$$ 2\left(1-\frac{1}{|G|} \right)= m - \frac{|X|}{|G|} = m - \frac{\sum_{i=1}^m |\text{Orb}_G(x_i)|}{|G|} = \sum_{i=1}^m \left( 1 - \frac{1}{|\text{Stab}_G(x_i)|} \right).$$
Since $|G| \geq 2$, we see that$$ 1 \leq 2\left(1-\frac{1}{|G|}\right) < 2,$$ whilst any stabilizer $\text{Stab}_G(x_i)$ must contain at least two points — the identity, and the relevant nontrivial rotation of which $x_i$ is a pole, and hence we have$${1\over2} \leq \left( 1 - \frac{1}{|\text{Stab}_G(x_i)|} \right) < 1.$$ Thus, $m$ must be equal to $2$ or $3$.
If $m = 2$, we see that$$2\left(1-\frac{1}{|G|}\right)=m - \frac{\sum_{i=1}^m |\text{Orb}_G(x_i)|}{|G|}$$ rearranges to $$|\text{Orb}_G(x_1)| + |\text{Orb}_G(x_2)| = 2$$and there are exactly two poles in $X$, which must be antipodal, and so $G$ consists of rotations about a specific axis of rotation, which passes through $x_1$ and $x_2$. Since $G$ is finite, one can show that $G$ is cyclic by considering the rotation in $G$ of least angle about the axis to show that it generates $G$ — show that any other rotation is by an angle which is an integer multiple of the least angle.