Let $A\in M_2(\mathbb{Z})$ with nonzero determinant. Show that there exist infinitely many numbers $s\in\mathbb{N}$ such that $$ \exists a_s\in\mathbb{N}^*:A^{a_s}-I \equiv O\mod s. $$
My attempt is to use prove by contradiction. Suppose that for all $x$ in a finite set of natural number and for all $a\ne 0$ natural number, we have $$ A^a-I\not\equiv O \mod x \Leftrightarrow A^a\not\equiv I \mod x. $$ Applying determinant to the above formula, we get that$$ a\det(A)\not\equiv 1 \mod x. $$ So, we get that $x$ not devide $a\det(A)-1$ or, for some $k\in\mathbb{Z}$, we have $$ a\det(A)-1\ne kx,\quad \forall x\in\{x_1,...,x_l\} \text{ and } a\in\mathbb{N}^*. $$ Moreover we get that$$ \det(A)\ne \frac{kx+1}{a},\quad \forall x\in\{x_1,...,x_l\} \text{ and } a\in\mathbb{N}^*. $$ Then letting $a$ go to infinity, we obtain $$ \det(A)\ne 0, $$ which not contradict the hypothesis. Does this argument really can work? Does exists other path to get the claim in the conclusion? Thanks in advanced!
Hints:
Let $\det A = d \ne 0$ (over $\mathbb Z$).