A "direct geometric argument" that Z(SO(3)) is the trivial group.

Question

My question pertains to a "direct geometric argument" that Z(SO(3)) is the trivial group, as articulated in Exercises 3.5.4 and 3.5.5 of John Stillwell's Naive Lie Theory. The first exercise establishes that no rotation about the $e_1$-axis that is neither the identity nor the half-turn commutes with a half-turn about the $e_3$-axis. The second exercise establishes that there exists a rotation (such as, say, the quarter-turn about the $e_3$-axis) that does not commute with the half-turn about the $e_1$-axis. These two facts obviously imply that no non-identity rotation about the $e_1$-axis belongs to $Z(SO(3))$. But my question is why this implies that $Z(SO(3))$ is trivial.