In our mathematics course, we have defined $\overline{a}=\{a \in \mathbb{Z} \mid k \equiv a \text{ mod } m \}=\frac{a-k}{m}$. We have also defined $\overline{a} \times \overline{b}=\overline{ab}$.
In order to calculate $\overline{ab}$, can I simply multiple together $\frac{a-k}{m}$ and $\frac{b-k}{m}$ to get $\frac{(a-k)(b-k)}{m^2}$?
My apologies if this question is too simplistic.
Assuming what definitions you've written above are written poorly, and what I am thinking of is indeed what you intended, I will clean up the definitions here below and show what and why the multiplication is well defined.
Recognize then that $\equiv\dots \pmod{m}$ is an equivalence relation on $\Bbb Z$
Recognize then that $\overline{a}=\overline{b}$ if and only if $a\equiv b\pmod{m}$
We define $\overline{a}\times \overline{b}$ to be equal to $\overline{(ab)}$. We wish to prove this is well defined. That is to say, we want to show that regardless how we represent the classes, the result is the same.
Suppose $\overline{a_1}=\overline{a_2}$ and $\overline{b_1}=\overline{b_2}$. We wish to prove that $\overline{(a_1b_1)}=\overline{(a_2b_2)}$
Since $a_1\equiv a_2\pmod{m}$ we have $(a_1-a_2)=xm$ for some integer $x$. Similarly $(b_1-b_2)=ym$ for some integer $y$.
We have $a_1b_1 - a_2b_2 = a_1b_1-\underbrace{(a_1-a_2-xm)}_{\text{equals 0}}b_1-a_2b_2-\underbrace{(b_1-b_2-ym)}_{\text{equals 0}}a_2$
$=a_1b_1-a_1b_1+a_2b_1+xmb_1-a_2b_2-a_2b_1+a_2b_2+yma_2$
$=xmb_1+yma_2 = (xb_1+ya_2)m$ which is an integer multiple of $m$
Therefore we do indeed have $\overline{(a_1b_1)}=\overline{(a_2b_2)}$
You ask how to calculate $\overline{ab}$ given an $a$ and a $b$. You simply multiply $a$ and $b$ together like normal and then put a line over top of it.
For example, suppose we are working in $\Bbb Z_{10}$. One has $\overline{5}\times \overline{82} = \overline{(5\cdot 82)}=\overline{410}$
Granted, this is not necessarily the most useful way of expressing the result, but it is none-the-less a correct way of expressing the result.
More often, however, we are interested in using canonical representatives of the equivalence classes, preferring only to refer to equivalence classes by the unique element in the range $[0,m)$. Here, we would have $\overline{5}\times \overline{82}=\overline{410}=\overline{0}$.
To find the canonical representative for the resulting equivalence class, one can run the quotient-remainder theorem on it. $\overline{ab}=\overline{(ab-m\lfloor\frac{ab}{m}\rfloor)}$