I am asked to show that the group $GL(2,\mathbb{Z}/p\mathbb{Z})$ is a group.
I am aware of how to show the group properties and have an idea of how to do this.
So, showing that an inverse exists will then imply that the group properties are satisfied.
Let for example $p = 11$
Let $$A = \left(\begin{matrix} 7 & 3 \\ -3 & 7 \\ \end{matrix}\right) $$
So we need to find a matrix $A^{-1}$ such that $AB=BA=I$. Then $$A^{-1} = \frac{1}{58}\left(\begin{matrix} 7 & -3 \\ 3 & 7 \\ \end{matrix}\right)$$
However, I am not able to get farther than this. I know I have to reduce this modulo 11, but not sure how to do it, as I am a bit rusty on modular arithmetic. Any hints or suggestions on how to do it will be most welcome.
Thank you.
Ignore the $\mathbb{Z}/p\mathbb{Z}$ and prove that $\text{GL}(2,k)$ is a group (under matrix multiplication) whenever $k$ is a field. Notice also that $\text{Gl}(2,k)$ is defined as the set of invertible $2\times 2$-matrices over $k$. In other words, there is nothing to prove!
Having said that though, let $\begin{pmatrix} a&b\\ c&d \end{pmatrix}\in \text{M}(2,k)$.Try to find $\begin{pmatrix} a&b\\ c&d \end{pmatrix}^{-1}$. Notice that you can use the same algorithms as for real matrices (why?). You will quickly discover that $\begin{pmatrix} a&b\\ c&d \end{pmatrix}$ is invertible if and only if $ad-bc\neq 0$ in $k$. In other words, if and only if $\det\begin{pmatrix} a&b\\ c&d \end{pmatrix}\neq 0$.
As for your example $A=\begin{pmatrix} 7&3\\ -3&7 \end{pmatrix}$. Indeed, $A^{-1}=\frac{1}{58}\begin{pmatrix} 7&-3\\ 3&7 \end{pmatrix}$. The question is, what is $\frac{1}{58}$? It helps to write it as $58^{-1}$. Clearly, $58^{-1}=3^{-1}$ in $\mathbb{Z}_{11}$. Notice that $3\cdot 4=12=1$ in $\mathbb{Z}_{11}$, thus $3^{-1}=4$. It follows that $A^{-1}=\begin{pmatrix} 6&10\\ 1&6 \end{pmatrix}.$
EDIT: Here is a general procedure of finding inverses in $\mathbb{Z}_n$. Let $\bar{x}\in \mathbb{Z}_n$ such that $\text{gcd}(x,n)=1$. By Bézout's theorem there exist integers $\alpha,\beta$ such that $\alpha x+\beta n=1$. Consider the last expression modulo $n$ yields $\alpha x\mod n=1$ hence $\bar{x}^{-1}=\bar{\alpha}$. You can find $\alpha,\beta$ explicitly using the Euclidean algorithm.