Given a Latin square
how would one tell if the operation is associative without trying every combination? Or is there something to look for that would at least limit the amount of combinations I have to try?
In this case I found $(bb)b \ne b(bb)$ but I struggled quite a bit.

In this particular case, it is easy to see that $e$ is an identity.
Lemma: Given a latin square with an identity, the operation is associative if and only if the latin square is the table of a group.
This is immediate. Indeed $\Leftarrow$ is obvious.
For $\Rightarrow$ associativity and identity are given, and it is immediate that latin square implies inverses.
\qed
For the given table it is trivial to see it is not coming from a group, as any group with 5 elements is cyclic thus commutative, and your square is not.
In general, given a larger latin square it may be hard to decide if it is the table of a group or not.