Here is Prob. 44, Sec. 8, in the book A First Course In Abstract Algebra by John B. Fraleigh, 7th edition:
In analogy with Examples 8.7 and 8.10, consider a regular plane $n$-gon for $n \geq 3$. Each way that two copies of such an $n$-gon can be placed, with one covering the other, corresponds to a certain permutation of the vertices. The set of these permutations is a group, the $n$th dihedral group $D_n$, under permutation multiplication. Find the order of this group $D_n$. Argue geometrically that this group has a subgroup having just half as many elements as the whole group has.
My Attempt:
Here is a somewhat related post of mine here on Math Stack Exchange.
Let our regular $n$-gon going counter-clockwise have vertices $$ P_r \left( \cos \frac{2r \pi}{n}, \sin \frac{2r \pi}{n} \right) \ \mbox{ for } \ r = 0, 1, \ldots, n-1. $$
Then we can, for each $k = 1, \ldots, n-1$, rotate the plane clockwise or counter-clockwise through an angle of $\frac{2k \pi}{n}$ radians obtain a regular $n$-gon covering our original $n$-gon, with vertices on top vertices (albeit with the vertices permuted of course). Thus, for each $r = 0, 1, \ldots, n-1$, the vertex $P_r$ moves to the point $R_{r,k}$, where $$ \begin{align} R_{r,k} &= \begin{cases} \left( \cos \frac{ 2(r+k) \pi }{n} , \sin \frac{ 2(r+k) \pi }{n} \right) \ & \mbox{ if the rotation is counter-clockwise}, \\ \left( \cos \frac{ 2(r-k) \pi }{n} , \sin \frac{ 2(r-k) \pi }{n} \right) \ & \mbox{ if the rotation is clockwise} \end{cases} \\ &= \begin{cases} \left( \cos \frac{ 2 \left( r +_n k \right) \pi }{n} , \sin \frac{ 2 \left( r +_n k \right) \pi }{n} \right) \ & \mbox{ if the rotation is counter-clockwise}, \\ \left( \cos \frac{ 2(r + n-k) \pi }{n} , \sin \frac{ 2(r+n-k) \pi }{n} \right) \ & \mbox{ if the rotation is clockwise} \end{cases} \\ &= \begin{cases} \left( \cos \frac{ 2 \left( r +_n k \right) \pi }{n} , \sin \frac{ 2 \left( r +_n k \right) \pi }{n} \right) \ & \mbox{ if the rotation is counter-clockwise}, \\ \left( \cos \frac{ 2\left( r +_n (n-k) \right) \pi }{n} , \sin \frac{ 2 \left( r +_n (n-k) \right) \pi }{n} \right) \ & \mbox{ if the rotation is clockwise} \end{cases} \\ &= \begin{cases} P_{r+_n k} \ & \mbox{ if the rotation is counter-clockwise}, \\ P_{ r +_n (n-k) } \ & \mbox{ if the rotation is clockwise}. \end{cases} \end{align} $$ Here $i+_nj$ is defined as follows: $$ i+_nj = \begin{cases} i+j \ & \mbox{ if } 0 \leq i+j < n \\ i+j-n \ & \mbox{ if } i+j > n. \end{cases} $$ for any $i, j \in \{ 0, 1, \ldots, n-1 \}$.
Rotation (clockwise or counter-clockwise) through the angle $0$ radians of course leaves the vertices of our $n$-gon unchanged.
These $n$ rotations of course form a subgroup of the group of the symmetries of the regular $n$-gon containing half as many elements as the whole group has.
The remaining $n$ elements are obtained as follows:
Now let P(a, b) be any point in the plane, let $m$ and $c$ be any real numbers such that $m \neq 0$, and let $$ \mathscr{L} \colon= \left\{ \ (x, y) \in \mathbb{R}^2 \, \vert \, y = mx+c \ \right\} $$ be the straight line in the plane with slope $m$ and $y$-intercept $c$. If point $P$ does not lie on line $\mathscr{L}$, then the mirror image of $P(a, b)$ into $\mathscr{L}$ is defined as the point $Q(s, t)$ in the plane such that
(1) the straight line $\mathscr{L}^\prime$ in the plane passing through points $P$ and $Q$ is perpendicular to line $\mathscr{L}$, and
(2) the midpoint $M \left( \frac{a+s}{2}, \frac{b+t}{2} \right)$ of the line segment $\overline{PQ}$ joining points $P$ and $Q$ lies on line $\mathscr{L}$.
These two conditions give us the following two simultaneous equations in $s$ and $t$: $$ \begin{align} \frac{t-b}{s-a} &= \frac{-1}{m}, \\ \frac{t+b}{2} &= m \frac{s+a}{2} + c. \end{align} $$
These last two equations can be rewritten as $$ \begin{align} t-b &= \frac{-1}{m} (s-a), \tag{1}\\ t+b &= m (s+a) + 2c. \tag{2} \end{align} $$
Upon adding together the last two equations we get $$ 2t = \left( m - \frac{1}{m} \right) s + \left( m + \frac{1}{m} \right) a + 2c, $$ and hence $$ t = \frac{1}{2} \left( m - \frac{1}{m} \right) s + \frac{1}{2} \left( m + \frac{1}{m} \right) a + c. \tag{3} $$
Now substituting the value of $t$ from (3) into (1) above we obtain $$ \frac{1}{2} \left( m - \frac{1}{m} \right) s + \frac{1}{2} \left( m + \frac{1}{m} \right) a + c - b = \frac{-1}{m} (s-a), $$ and upon simplifying we obtain $$ \frac{1}{2} \left( m + \frac{1}{m} \right) s + \frac{1}{2} \left( m - \frac{1}{m} \right) a + c - b = 0, $$ which yields $$ s = - \frac{ \frac{1}{2} \left( m - \frac{1}{m} \right) a + c - b }{ \frac{1}{2} \left( m + \frac{1}{m} \right) } = \frac{ 2(b-c)m + \left( 1 - m^2 \right)a }{ 1 + m^2 }. \tag{4} $$
Putting the value of $s$ from (4) into (3) above, we find that $$ \begin{align} t &= \frac{1}{2} \left( m - \frac{1}{m} \right) \left( - \frac{ \frac{1}{2} \left( m - \frac{1}{m} \right) a + c - b }{ \frac{1}{2} \left( m + \frac{1}{m} \right) } \right) + \frac{1}{2} \left( m + \frac{1}{m} \right) a + c \\ &= \frac{ m^2 - 1}{2m} \frac{ 2(b-c)m + \left( 1 - m^2 \right)a }{ 1 + m^2 } + \frac{ \left( m^2+1 \right) a }{2m} + c \\ &= \frac{ \left( m^2 + 1 \right)^2 a - \left(m^2 - 1 \right)^2 a + 2m \left( m^2 - 1 \right) b + 2m \left( 1 - m^2 \right)c + 2m \left( 1 + m^2 \right) c }{2m \left( 1 + m^2 \right) } \\ &= \frac{ 4m^2 a + 2m \left( m^2 - 1 \right) b + 4mc }{ 2m \left( 1 + m^2 \right) }. \end{align} \tag{5} $$
Thus we obtain the point $Q(s, t)$, where $$ \begin{align} s &= \frac{ 2(b-c)m + \left( 1 - m^2 \right)a }{ 1 + m^2 }, \\ t &= \frac{ 4m^2 a + 2m \left( m^2 - 1 \right) b + 4mc }{ 2m \left( 1 + m^2 \right) } \\ &= \frac{ 2m a + \left( m^2 - 1 \right) b + 2c }{ 1 + m^2 }. \end{align} \tag{5} $$
Therefore the mirror image of the point $P(a, b)$ into the line $$ \mathscr{L} = \left\{ \ (x, y) \in \mathbb{R}^2 \, y = mx + c \, \right\}, $$ where $m$ and $c$ are any real numbers such that $m \neq 0$, is the point $$ Q \left( s, t \right) = Q \left( \frac{ 2(b-c)m + \left( 1 - m^2 \right)a }{ 1 + m^2 }, \frac{ 2m a + \left( m^2 - 1 \right) b + 2c }{ 1 + m^2 } \right). \tag{A} $$ And, these two points will of course coincide if $P(a, b)$ lies on $\mathscr{L}$.
For $c = 0$, we find that the mirror image of the point $P(a, b)$ into the line $$ \mathscr{L} = \left\{ \ (x, y) \in \mathbb{R}^2 \, y = mx \, \right\}, $$ where $m$ is any real numbers such that $m \neq 0$, is the point $$ Q \left( s, t \right) = Q \left( \frac{ 2bm + \left( 1 - m^2 \right)a }{ 1 + m^2 }, \frac{ 2m a + \left( m^2 - 1 \right) b }{ 1 + m^2 } \right). \tag{A*} $$
If $m = 0$, then the mirror image of the point $P(a, b)$, where $a, b \in \mathbb{R}$ and $b \neq 0$, into the $x$-axis is the point $Q(a, -b)$, because (1) the line through $P$ and $Q$ is perpendicular to the $x$-axis, and (2) the midpoint $M(a, 0)$ of the line segment $\overline{PQ}$ lies on the $x$-axis.
Similarly, the mirror image of the point $P(a, b)$, where $a, b \in \mathbb{R}$ and $a \neq 0$, into the $y$-axis is the point $Q(-a, b)$.
Thus the mirror image of the point $P(a, b)$ into a line $\mathscr{L}$ passing through the origin but not through the point $P$ itself is the point $$ \begin{cases} Q \left( \frac{ 2bm + \left( 1 - m^2 \right)a }{ 1 + m^2 }, \frac{ 2m a + \left( m^2 - 1 \right) b }{ 1 + m^2 } \right) \ & \mbox{ if } \mathscr{L} \ \mbox{ has a finite, non-zero slope}, \\ Q(a, -b) \ & \mbox{ if } \mathscr{L} \ \mbox{ is the $x$-axis}, \\ Q(-a, b) \ & \mbox{ if } \mathscr{L} \ \mbox{ is the $y$-axis}. \end{cases} \tag{B} $$
With this preparation, we now return to our original problem.
Putting $m = \tan \frac{\pi}{n}$, $a = \cos \frac{2r \pi}{n}$, and $b = \sin \frac{2r\pi}{n}$ in the first of the three cases in (B) above, we note that, for each $r = 0, 1, \ldots, n-1$, the mirror image of the point $$ P_r \left( \cos \frac{ 2r \pi}{n}, \sin \frac{ 2r \pi}{n} \right) $$ into the line $$ \mathscr{L} = \left\{ \ (x, y) \in \mathbb{R}^2 \, \vert \, y = x \tan \frac{\pi}{n} \ \right\} $$ is the point $$ \begin{align} & \quad Q \left( \frac{ 2 \sin \frac{2r\pi}{n} \tan \frac{\pi}{n} + \left( 1 - \tan^2 \frac{\pi}{n} \right) \cos \frac{2r \pi}{n} }{ 1 + \tan^2 \frac{\pi}{n} }, \frac{ 2 \cos \frac{2r\pi}{n} \tan \frac{\pi}{n} - \left( 1 - \tan^2 \frac{\pi}{n} \right) \sin \frac{2r \pi}{n} }{ 1 + \tan^2 \frac{\pi}{n} } \right) \\ &= Q \left( \sin \frac{2r \pi}{n} \sin \frac{2 \pi}{n} + \cos \frac{2 \pi}{n} \cos \frac{2r \pi}{n} , \cos \frac{2r \pi}{n} \sin \frac{2 \pi}{n} - \cos \frac{2 \pi}{n} \sin \frac{2r \pi}{n} \right) \\ &= Q \left( \cos \frac{ 2(1-r ) \pi }{n}, \sin \frac{ 2(1-r ) \pi }{n} \right) \\ &= \begin{cases} Q \left( \cos \frac{ 2(1-r ) \pi }{n}, \sin \frac{ 2(1-r ) \pi }{n} \right) \ & \mbox{ for } r = 0, 1, \\ Q \left( \cos \frac{ 2(n+1-r ) \pi }{n}, \sin \frac{ 2(n+1-r ) \pi }{n} \right) \ & \mbox{ for } r = 2, \ldots n-1 \end{cases} \\ &= \begin{cases} P_1 \ & \mbox{ for } r = 0, \\ P_0 \ & \mbox{ for } r = 1, \\ P_{n+1-r } \ & \mbox{ for } r = 2, \ldots, n-1. \end{cases} \end{align} $$ Thus reflection through the line through the origin making an angle of $\frac{\pi}{n}$ radians with the positive $x$-axis leaves the $n$-gon covering itself with vertices on top of vertices, thus giving us a symmetry of the $n$-gon.
More generally, for each $k = 1, \ldots, n-1$ such that $k \neq \frac{n}{2}$, we find that, for each $r = 0, 1, \ldots, n-1$, the mirror image of the point $$ P_r \left( \cos \frac{2r\pi}{n}, \sin \frac{2r\pi}{n} \right)$$ into the line $$ \mathscr{L} = \left\{ \ (x, y) \in \mathbb{R}^2 \, \vert \, y = x \tan \frac{k\pi}{n} \ \right\} $$ is the point $$ \begin{align} & \quad Q_r \left( \frac{ 2 \sin \frac{2r\pi}{n} \tan \frac{k\pi}{n} + \left( 1 - \tan^2 \frac{k\pi}{n} \right) \cos \frac{2r \pi}{n} }{ 1 + \tan^2 \frac{k\pi}{n} }, \frac{ 2 \cos \frac{2r\pi}{n} \tan \frac{k\pi}{n} - \left( 1 - \tan^2 \frac{k\pi}{n} \right) \sin \frac{2r \pi}{n} }{ 1 + \tan^2 \frac{k\pi}{n} } \right) \\ &= Q_r \left( \sin \frac{2r \pi}{n} \sin \frac{2k \pi}{n} + \cos \frac{2k \pi}{n} \cos \frac{2r \pi}{n} , \cos \frac{2r \pi}{n} \sin \frac{2k \pi}{n} - \cos \frac{2k \pi}{n} \sin \frac{2r \pi}{n} \right) \\ &= Q_r \left( \cos \frac{ 2(k-r ) \pi }{n}, \sin \frac{ 2(k-r ) \pi }{n} \right) \\ &= \begin{cases} Q_r \left( \cos \frac{ 2(k-r ) \pi }{n}, \sin \frac{ 2(k-r ) \pi }{n} \right) \ & \mbox{ for } r = 0, \ldots, k, \\ Q_r \left( \cos \frac{ 2(n+k-r ) \pi }{n}, \sin \frac{ 2(n+k-r ) \pi }{n} \right) \ & \mbox{ for } r = k+1, \ldots, n-1 \end{cases} \\ &= \begin{cases} P_{k-r} \ & \mbox{ for } r = 0, \ldots, k, \\ P_{n+k-r } \ & \mbox{ for } r = k+1, \ldots, n-1. \end{cases} \end{align} $$ Thus reflection through the line through the origin making an angle of $\frac{k\pi}{n}$ radians with the positive $x$-axis, for each $k = 1, \ldots, n-1$ such that $k \neq \frac{n}{2}$, leaves the $n$-gon covering itself with vertices on top of vertices, thus giving us a symmetry of the $n$-gon.
If $n$ is even and $k = \frac{n}{2}$, then $\frac{k\pi}{n} = \frac{\pi}{2}$, and then using the third case in (B) above we find that, for each $r = 0, 1, \ldots, n-1$, the mirror image of the point $$ P_r \left( \cos \frac{2r\pi}{n}, \sin \frac{2r\pi}{n} \right)$$ into the $y$-axis is the point $$ \begin{align} &\quad Q_r \left( -\cos \frac{2r\pi}{n}, \sin \frac{2r\pi}{n} \right) \\ &= Q_r \left( \cos \frac{(n-2r)\pi}{n}, \sin \frac{(n-2r)\pi}{n} \right) \\ &= Q_r \left( \cos \frac{ 2\left( \frac{n}{2} - r \right) \pi}{n}, \sin \frac{2\left( \frac{n}{2} - r \right)\pi}{n} \right) \\ &= Q_r \left( \cos \frac{ 2(k-r)\pi }{n} , \sin \frac{ 2(k-r)\pi }{n} \right) \\ &= \begin{cases} P_{k-r} \ & \mbox{ for } r = 0, 1, \ldots, k, \\ P_{n+k-r} \ & \mbox{ for } r = k+1, \ldots, n-1. \end{cases} \end{align} $$ Thus if $n$ is even, then reflection through the $y$-axis leaves the $n$-gon covering itself with vertices on top of vertices, thus giving us a symmetry of the $n$-gon.
Finally, for $k = 0$, we use the second of the three cases in (B) above, and conclude that, for each $r = 0, 1, \ldots, n-1$, the mirror image of the point $$ P_r \left( \cos \frac{2r\pi}{n}, \sin \frac{2r\pi}{n} \right)$$ into the $x$-axis is the point $$ \begin{align} &\quad Q_r \left( \cos \frac{2r\pi}{n}, - \sin \frac{2r\pi}{n} \right) \\ &= Q_r \left( \cos \frac{-2r\pi}{n}, \sin \frac{-2r\pi}{n} \right) \\ &= \begin{cases} P_0 \ & \mbox{ for } r = 0, \\ P_{n-r} \ & \mbox{ for } r = 1, \ldots, n-1. \end{cases} \end{align} $$ Thus reflection through the $x$-axis leaves the $n$-gon covering itself with vertices on top of vertices, thus giving us a symmetry of the $n$-gon.
Through what I have done so far, have I managed to describe the phenomenon Fraleigh has mentioned completely and rigorously enough? Or, are there details that still need to be included?
Or, is my analysis incorrect or erroneous in any detail(s)?
It seems to me that the author expects a less formula-heavy solution. Once you know the answer to the second question, computing the order of the group is easier, because you only need to count the number of symmetries in that subgroup.
Here are two increasingly revealing hints for the second question:
Hint 1: Label each vertex of the $n$-gon with a natural number from $1$ to $n$, say, clockwise. Some symmetries will preserve this orientation (the numbers will remain ordered clockwise), but there will also be some other symmetries inverting it.
Hint 2: What happens to orientation-preserving symmetries if you look at them through a mirror?