Define
$$M=\left\{\begin{bmatrix}a_{11}&a_{12}\\a_{21}&a_{22}\end{bmatrix}\in(\Bbb Z/13)_{22}:a_{12}=0\right\}\;.$$
(This is the set of $2\times 2$ matrices with integers mod $13$ with the upper right entry $= 0$.)
(a) Show that $M$ forms a submonoid of $\big((\Bbb Z/13)_{22},\cdot,I_2\big)$.
(b) Determine the order $|M^*|$ of the group $M^*$ of units of $M$.
I'm having trouble with the closure under multiplication part. For example if you multiply two matrices, $A$ and $B$ of the set $M$, then in the first entry, you get $a_{11}b_{11}$, which equals some number $c$. But, this number $c$ is not necessarily an element of the integers mod $13$, $\Bbb Z/13$, is it? So, how is $M$ closed under multiplication?
For the second question, a $2\times 2$ matrix is invertible as long as $ad-bc$ is nonzero, so we just need that $a_{21}$ is nonzero. So, then what would the order be?
The multiplication takes place in $\Bbb Z/13$: for example, $5\cdot6=4$, since $30\bmod 13=4$. The only thing that you actually have to check is that $(AB)_{12}=0$ whenever $A,B\in M$.
For invertibility be a bit careful:
$$\begin{bmatrix}a_{11}&0\\a_{21}&a_{22}\end{bmatrix}\begin{bmatrix}b_{11}&0\\b_{21}&b_{22}\end{bmatrix}=\begin{bmatrix}1&0\\0&1\end{bmatrix}$$
if and only if
$$\begin{align*} a_{11}b_{11}&=1\\ a_{21}b_{11}+a_{22}b_{21}&=0\\ a_{22}b_{22}&=1\;, \end{align*}$$
where the arithmetic is all done mod $13$. The first and third equations tell you that the diagonal elements of $A$ and $B$ must be non-zero, and since $13$ is prime, every non-zero element of $\Bbb Z/13$ has a multiplicative inverse. Thus, you can choose $a_{11}$ and $a_{22}$ to be any non-zero elements of $\Bbb Z/13$. Doing so will determine $b_{11}$ and $b_{22}$ completely, so the question is then how many solutions the equation $a_{21}b_{11}+a_{22}b_{21}=0$ has for $a_{21}$ and $b_{21}$. Again, remember that you’re working in $\Bbb Z/13$, not in $\Bbb Z$. More generally, if $a$ and $b$ are non-zero elements of $\Bbb Z/13$, how many solutions does $ax+by=0$ have in $\Bbb Z/13$? HINT: multiply the equation by $a^{-1}$.