I have two sets of triangles:
$A = \{[(0,0),(1,0),(1,1)], [(1,0),(1,1),(2,1)], [(2,1),(1,0),(2,0)]\}$
and
$B = \{[(1,0),(0,0),(1,1)], [(2,1),(1,0),(2,0)], [(2,1),(1,1),(1,0)]\}$
What algorithm can be used to sort (and what would it order by) for the purpose of comparison? In this case both are the same but I want to know also when they are not the same.
Simply sort lexicographically. In general any consistent ordering method would work if you wish to compare equality.