I need to prove the following operation is associative:
$x*y = xy \pmod 5$
I came up with the equation that $x*y=xy-5[\![xy/5]\!]$
I'm having difficulty proving that $x*(yz)=(xy)*z$. After expanding and then trying to simplify, I have that $x*(yz)=xyz-5z[\![xy/5]\!]$ and $(x*y)*z=xyz-5x[\![yz/x]\!]$ , but cannot then show that the two are equal.
Try to build up $(x*y)*z$ and $x*(y*z)$ in small steps:
From your description we have $xy = 5a + (x*y)$ and $yz = 5b + (y*z)$ for suitable integers $a$ and $b$ (these were the values of the $\mathrm{floor}$ function but that is not really important here).
When you calculate
$$(x*y)z = (xy - 5a)z$$ and $$x(y*z) = x(yz - 5b)$$
you will find that both have the same remainder $\mod 5$.