I am unsure about the clarity of my explanation of the finitude of the group.
We have a matrix: $$\mathbf{M}=\begin{bmatrix}\cos(2\pi/m)&-\sin(2\pi/m)\cr \sin(2\pi/m)&\cos(2\pi/m)\end{bmatrix}$$ We also have a $2\times 2$ matrix sequence $\lbrace \mathbf{X_0},\mathbf{X_1},\mathbf{X_2}\dots\rbrace$ with the formula: $$\mathbf{X_k}=\mathbf{P}^k(\mathbf{X_0}-\mathbf{B})+\mathbf{B}\qquad\qquad k\geq0$$ where $\mathbf{B}=(\mathbf{I}-\mathbf{P})^{-1}\mathbf{Q}$ and $\mathbf{P},\mathbf{Q}\in\mathbb{R}^{2\times2}$, and a binary operation $\circ$ which is defined as follows: $\mathbf{X_i}\circ\mathbf{X_j}$ is the result of replacing $\mathbf{X_j}$ in the expression of $\mathbf{X_i}$.
Show that if $\mathbf{P}=\mathbf{M}$ then the set $G=\lbrace\mathbf{X_1},\mathbf{X_2},\dots\rbrace$ forms a finite group under $\circ$.
First remarks
$\mathbf{M}$ describes a rotation anticlockwise about the origin of $\frac{2\pi}{m}$ so $\mathbf{M}^m$ describes a rotation of $2\pi$ (or in other words, no rotation at all); therefore, $\mathbf{M}^m=\mathbf{I}$.
We can work out $\mathbf{X_i}\circ\mathbf{X_j}$: $$ \begin{align*} \mathbf{X_i}\circ\mathbf{X_j}&=\mathbf{M}^i(\mathbf{X_j}-\mathbf{B})+\mathbf{B} \cr &=\mathbf{M}^i(\mathbf{M}^j(\mathbf{X_0}-\mathbf{B})+\mathbf{B}-\mathbf{B})+\mathbf{B}\cr &=\mathbf{M}^{i+j}(\mathbf{X_0}-\mathbf{B})+\mathbf{B} \end{align*} $$
Notice that if we similarly computed $\mathbf{X_j}\circ\mathbf{X_i}$, we would get the same expression as above (just with $j+i$ in the exponent which makes no difference) so $\circ$ commutes.
Axioms
Identity
Consider: $$ \begin{align*} \mathbf{X_i}\circ\mathbf{X_m}&=\mathbf{M}^{i+m}(\mathbf{X_0}-\mathbf{B})+\mathbf{B}\cr &=\mathbf{M}^{i}(\mathbf{X_0}-\mathbf{B})+\mathbf{B}\cr &=\mathbf{X_i} \end{align*} $$ However $\circ$ commutes so $\mathbf{X_i}\circ\mathbf{X_m}=\mathbf{X_m}\circ\mathbf{X_i}=\mathbf{X_i}$ so we can see that $\mathbf{X_m}$ is the identity element in $G$ as it displays left- and right-identity.
Inverses
Consider: $$ \begin{align*} \mathbf{X_i}\circ\mathbf{X_{m-i}}&=\mathbf{M}^{i+(m-i)}(\mathbf{X_0}-\mathbf{B})+\mathbf{B}\cr &=\mathbf{M}^m(\mathbf{X_0}-\mathbf{B})+\mathbf{B}\cr &=\mathbf{X_m} \end{align*} $$
which is the identity of $G$. Therefore for all $\mathbf{X_i}\in G$ there exists an inverse element $\mathbf{X_{m-i}}\in G$.
Closedness
Consider the following:
$$ \mathbf{X_i}\circ\mathbf{X_j}=\mathbf{M}^{i+j}(\mathbf{X_0}-\mathbf{B})+\mathbf{B} $$
We can use the division algorithm to write $i+j=qm+r$ where $q\geq0$ and $0 \leq r < m$, which is equivalent to: $$\mathbf{M}^{i+j}=(\mathbf{M}^{m})^q\mathbf{M}^r=\mathbf{M}^r$$ but since $\mathbf{M}^0=\mathbf{M}^m=\mathbf{I}$ we can rewrite $r$'s interval as $0< r \leq m$. Therefore: $$\mathbf{X_i}\circ\mathbf{X_j}=\mathbf{M}^r(\mathbf{X_0}-\mathbf{B})+\mathbf{B}=\mathbf{X_r}$$
Due to $r$'s interval, $X_r\in G$ so $G$ is closed under $\circ$.
Moreover $G$ is finite as we can write $\mathbf{M}^b=\mathbf{M}^{mq+r}$ for any $b\in\mathbb{Z}^+$ using the division algorithm, and then say $\forall r\in(0,m]\,\,\mathbf{M}^{mq+r}=\mathbf{M}^r$ so $\mathbf{X_{b}}=\mathbf{X_{qm+r}}=\mathbf{X_r}$. However the set $G$ must contain unique elements so $G=\lbrace\mathbf{X_k}\rbrace_{k=1}^m$ and is thus finite. (not sure how to make this clearer)
Associativity
Requires trivial algebraic manipulation to show that $(\mathbf{X_i}\circ\mathbf{X_j})\circ\mathbf{X_k}=\mathbf{X_i}\circ(\mathbf{X_j}\circ\mathbf{X_k})$ so $\circ$ is associative.
Since all four group axioms are fulfilled and $G$ is shown to be finite, $(G,\circ)$ is a finite group. $\blacksquare$