I'm reading A First Course in Modular Forms by Diamond and Shurman and am confused on a small point in Chapter 2. Let $\Gamma$ be a congruence subgroup of $\operatorname{SL}_2(\mathbb Z)$. $\gamma \in \mathscr H$ is called an elliptic point for $\Gamma$ if the stablizer of $\gamma$ in $\operatorname{PSL}_2$ is nontrivial.
Proposition 2.1.1 Let $\tau_1, \tau_2 \in \mathscr H$ be given. There exist open neighborhoods $U_i$ of $\tau_i$ in $\mathscr H$ such that if $\gamma \in \operatorname{SL}_2(\mathbb Z), \gamma(U_1) \cap U_2 \neq \emptyset$, then $\gamma(\tau_1) = \tau_2$.
Corollary 2.2.3 Let $\Gamma$ be a congruence subgroup of $\operatorname{SL}_2(\mathbb Z)$. Each point $\tau \in \mathscr H$ has a neighborhood $U$ in $\mathscr H$ such that $\gamma \in \Gamma, \gamma(U) \cap U \neq \emptyset$ implies $\gamma \in \operatorname{Stab} \tau$. Such a neighborhood has no elliptic points except possibly $\tau$.
Taking $\tau = \tau_1 = \tau_2$ and $U = U_1 \cap U_2$ in the proposition implies everything in the corollary except for the last sentence. How do we know that we can choose $U$ small enough to exclude all elliptic points? In other words, how do we know that the elliptic points in $\mathscr H$ form a discrete set?
Let $\tau$ and $U$ be as in the Corollary. If $\tau' \in U$ is an elliptic point different from $\tau$, with nontrivial stabilizer $\gamma \in \text{PSL}_2(\mathbb{Z})$, then $\gamma$ also fixes $\tau$ by the first part of the Corollary. It is thus sufficient to show that if $\gamma \in \text{PSL}_2(\mathbb{R})$ fixes two distinct points of $\mathscr{H}$, then $\gamma$ is the identity. This is immediately verified (say, by assuming that one of the points is $\sqrt{-1}$).