I just had this question Show that for all $k,l\in \mathbb Z, f(k+l)=f(k)+f(l)+kl$, that $f(0)=0$ and that for all $k\in \mathbb Z\,, f(-k)=k^2-f(k)$
$$\text{Let }\quad f:\mathbb Z \to \mathbb Z \quad \text{ and }\quad M_f:\mathbb Z \to \mathbb R^{3,3}, \quad k\mapsto \begin{bmatrix} 1 & k & f(k) \\ 0 & 1 & k \\ 0 & 0 & 1 \end{bmatrix} .$$ Further let $U:=\operatorname{image}(M_f)$. The function $f$ is chosen such that for all $A,B \in U$ $AB\in U$.
Then it was shown that for all $k,l\in \mathbb Z, f(k+l)=f(k)+f(l)+kl$, that $f(0)=0$ and that for all $k\in \mathbb Z\,, f(-k)=k^2-f(k)$.
$ \rule{18cm}{0.4pt}$
Now I want to show that $U$ is group in respect to matrix multiplication.
$\bullet$ Assoziativity is given by associativity of matrix multiplication, i..e $M_f(M_gM_h)=(M_fM_g)M_h$
$\bullet$ the neutral element is the identity matrix, but here has to be something wrong because if the identity matrix is the neutral element in this group. Then the inverse element has to be a lower triangular matrix but this is $\notin U$.
Some help is welcome!
$\bullet$ inverse element is $M_{f(-k)}$, since $ M_{f(k)}\cdot M_{f(-k)}= \begin{bmatrix} 1 & k & f(k) \\ 0 & 1 & k \\ 0 & 0 & 1 \end{bmatrix}\begin{bmatrix} 1 & -k & -f(k)+k^2 \\ 0 & 1 & -k \\ 0 & 0 & 1 \end{bmatrix}=I_n$
$\bullet$ neutral element is $I_n$
Thus we have for all $A,B,C \in U$: $(AB)C=A(BC)$, $A\in U \Rightarrow A^{-1} \in U$ with $AA^{-1}=I_n$ and $A\cdot I=A=I_n\cdot A \in U$. Therefore $U$ is a group.