How do I check that my edge normals in an arbitary triangle are inward (or outward)?

Question

I am considering an arbitary 2D triangle with points:

X1=(x1,y1), X2=(x2,y2), X3=(x3,y3).

I am then trying to compute the inward (or outward) normals to each edge such that they are all facing inward (or outward) but am strugling here.

Currently I have my normals by multiplying by a rotation matrix R=[0 -1;1 0] to give:

n12=R*(X1-X2)

n13=R*(X1-X3)

n32=R*(X3-X2)

but I have no idea if they are all facing inwards for an arbitary triangle.

The reason I need to keep this arbitary is so that I can compute this for multiple triangles in a mesh.

Any help will be appreciated.

Answer 1

To expand on Rahul’s comments, before you can talk about “inward” and “outward” normals, you first have to determine where the inside and outside of the triangle lie. This is equivalent to figuring out whether traversing the vertices in numerical order corresponds to a clockwise or counterclockwise orientation of the vertices. So, examine the sign of the determinant $$\Delta = \begin{vmatrix}x_1&y_1&1\\x_2&y_2&1\\x_3&y_3&1\end{vmatrix}.$$ If it is positive, then the vertices are in a counterclockwise arrangement; if negative, they are clockwise. (If the determinant vanishes, then the points are colinear—the triangle is degenerate.)

You can now find inward normals via a 90-degree rotation as in your question, but to do this you need to be careful to use the same orientation for all of the difference vectors. In your question, you use $X_1-X_2$ and $X_1-X_3$, which traverse the edges of the triangle in opposite directions. If you apply the same rotation to both of these, you will end up with one inward and one outward normal no matter which direction you rotate them in. For the three edge vectors to have consistent orientation, you need to take these differences in a consistent order: $X_2-X_1$, $X_3-X_2$ and $X_1-X_3$.

Your rotation matrix represents a counterclockwise 90-degree rotation, so if the triangle is oriented counterclockwise, multiplying each of the above differences by $R$ will produce inward normals; if it’s clockwise they will be outward, and you can get inward normals by negating them. In short, $$\begin{align} n_{12} &= \operatorname{sgn}(\Delta)R(X_2-X_1) \\ n_{23} &= \operatorname{sgn}(\Delta)R(X_3-X_2) \\ n_{31} &= \operatorname{sgn}(\Delta)R(X_1-X_3) \end{align}$$ produces inward normals regardless of the order of the vertices.

Answer 2

The cross product of two vectors produces another vector perpendicular (aka "normal") to both. The "sign" of the resulting vector swaps as you swap the order of the vectors.

Now, you get $$v12=\{x2-x1, \quad y2-y1, \quad z2-z1\}$$ and $$v13=\{x3-x1, \quad y3-y1, \quad z3-z1\}$$ The cross-product $n= v12$ x $v13$ heads to the same direction as the right-hand rule from $v12$ to $v13$. And the other normal (opposite) is $no= v13$ x $v12$

You are in a $2D$ case (set all $zi=0$), so knowing if $v12$ towards $v13$ is clockwise or counter-clockwise is easy:

$$res = (y2 - y1)·(x3 - x1) - (x2 - x1)·(y3 - y1)$$

will give a positive value if $(x3,y3)$ is at right of line $X1 \to X2$ wich means that $X1,X2,X3$ order is clockwise.