What formula will tell if three vertices in 3d space are ordered clockwise or counter-clockwise from the point of view of a camera?

Question

Assuming 3 ordered vertices in 3d space and a camera looking toward those points. What formula will tell me if they are seen clockwise or counter-clockwise in relation to their order?

Answer 1

Assuming the camera is at the origin, consider the $3\times 3$ matrix with column vectors equal to the three point vectors. The sign of the determinant of this matrix tells you the orientation.

Answer 2

Call the three points $p_1,p_2,p_3$, and assume the camera is located at the origin. One thing you could do is define $$N:=(p_2-p_1)\times(p_3-p_2),$$ where $\times$ denotes cross product, and then calculate $$S:=N\cdot p_1\quad.$$ If the plane spanned by the three points passes through the origin, you will get $S=0.$ Otherwise, the sign of $S$ will answer your question.