Also the binary operator $*$, describes this behavior:
$$\frac{k}{n} * \frac{l}{n} = \left\{\begin{matrix} \frac{k+l}{n} & \text{whenever} & k+l < n\\ & & \\ \frac{k+l-n}{n} & \text{whenever} & k+l\geq n \end{matrix}\right.$$
When I do the calculations I get 0 as the identity element but I can not find any inverses.
$$\frac{k}{n} * \frac{l}{n} = \left\{\begin{matrix} \frac{k+l}{n} & \text{whenever} & k+l < n\\ & & \\ \frac{k+l-n}{n} & \text{whenever} & k+l\geq n \end{matrix}\right.$$
The identity is $0$ since $0=\frac{0}{n}$ and $\frac{0}{n}*\frac{l}{n} = \frac{0+l}{n}=\frac{l}{n}$ since $0+l<n$; in fact, $\frac{l}{n} *0=\frac{l}{n} $ since $+$ is commutative.
Given $\frac{l}{n} $ where $0\le l\le n-1$, we want to find an inverse $(\frac{l}{n} )^{-1}$. That is, we want $p$ such that $\frac{l}{n} *\frac{p}{n}=0=\frac{p}{n}* \frac{l}{n} $ but then clearly $p=n-l$ since $$\frac{l}{n} *\frac{n-l}{n} =\frac{l+(n-l)-n}{n} =\frac{0}{n}$$
since $l+(n-l)\ge n$, indeed $\frac{n-l}{n} *\frac{l}{n}=0$ as well (since $+$ is commutative).
Lastly we prove closure. If $k+l<n$ then $\frac{k}{n}*\frac{l}{n}=\frac{k+l}{n}$ and $0\ge k+l<n$ so that $\frac{k+l}{n}\in G$. And if $k+l\ge n$ then because the maximum of these two is $n+n$ if we subtract $n$ from this under the second rule then $0\le k+l-n\le 2n - n= n$
Let me know if you need any further clarification!