There are two exercises in Stillwell's Naive Lie Theory that I'm having trouble doing:
3.8.4: Show that the subgroup $H = \{ \cos 2\pi r + i\sin 2\pi r : r\text{ rational} \}$ of the circle $SO(2)$ is totally disconnected but dense, that is, each arc of the circle contains an element of $H$.
3.8.5: Explain why there is no normal, dense, totally disconnected subgroup of $SO(n), n > 2$.
It feels obvious that $H$ is totally disconnected, but I'm not sure how to show that. I also haven't understood what Stillwell means by a subgroup being "dense". Is there a proper formal definition? I haven't studied topology or analysis yet, so maybe this is why I'm not getting this. (this book only requires linear algebra and a bit of calculus)
Since these are questions from a textbook, I'll give a hint (as opposed to a complete solution).
In your setting, you can use the following definition of dense: A subset $X$ of $\mathrm{SO}(n)$ is dense if any point in $\mathrm{SO}(n)$ is a limit of a sequence in $X$. You can use the fact that $\mathbb{Q}$ is dense in $\mathbb{R}$ to show that $H$ is dense in $\mathrm{SO}(2)$. To show that $H$ is totally disconnected you could use the intermediate value theorem or use the fact that the image of any path joining two distinct points must contain an arc of positive length, and then use the fact that the irrationals are dense in $\mathbb{R}$.
For the second question, you can proceed as follows. Suppose $H$ is a normal, totally disconnected subgroup of $\mathrm{SO}(n)$ for $n > 2$. Then you can use a minor modification of the proof in Section 3.8 (of the centrality of discrete normal subgroups) to show that $H$ is contained in the center of $\mathrm{SO}(n)$. But you know the center of $\mathrm{SO}(n)$ (from Section 3.7 of the text). Can any subgroup of the center be dense?