I want to categorize partial trimeshes (actually regions on trimeshes) based on whether they are inside out or outside out. (We can assume consistent orientation.) Above is an example of what I am looking for (normals in blue): P is inside out, Q is outside out. (All examples will be in 2D - the reasoning should transfer frictionlessly to 3D.)
I cannot use the standard test used to check if complete trimeshes are inside out: Both P and Q could be completed to either an inside-out or an outside-out polygon.
My current approch is to use the sum of angles between triangles (defined as depicted above, i.e. co-planar is 0°, convex is positive, concave is negative). The idea is: If the sum is negative, the polygon should be inside out, if it is negative outside out. I'm already taking into account the below situation (identical angles, but due to triangle size one is inside out, one is outside out) by weighting the sum by multiplying each angle by the added area of the incident triangles:
The problem with this is that it depends on the triangulation as shown below: Although the shapes are identical, when triangulated like on the left, not only do the negative angles get more weight than the positive one - there are two, so the sum is negative. Contrast the right triangulation: The negative angles are incident to so small triangles, that the strongly weighted positive angle dominates and the trimesh counts as outside out.
Can my approach be saved? Is there a better approach?
I'm more interested in a definition that's fast to compute, even if it should produce counter-intuitive results in corner cases. (Of all examples given, the ones I'm interested in look mostly like P.)