I'm trying to figure out how, given four arbitrary points in the plane, you can deterministically select points 1, 2, 3, and 4 where the union of the triangles 1-2-4 and 2-3-4 form a quadrilateral. Manually checking all 6 cases could work, but I'd like to find a nicer way of doing it.
Some notes:
- The triangles have clockwise winding order. That is, if there is a triangle in the XY plane where the vertices are oriented clockwise, the normal vector points in the positive Z direction.
- The quadrilateral may be concave.
- The algorithm should map any permutation of the vertices to the same quadrilateral.