Invert a triangle from three vectors (2D)

Question

How would I go, if I need to invert the area of a triangle, in 2D. Need this logic for a game. As I'm trying to mark everything but the area of the triangle.

I hope enough info is provided.

Answer 1

If you know some vectors and linear algebra you can transform the point you wonder if inside into the coordinate system with center in one of corners of triangle and unit axes pointing at the other corners.

Now if any one of the coordinates in the new coordinate system is negative OR the sum >1 then you are outside the triangle.

So if we have one vector pointing to one corner $\bf o$. Then two other vectors pointing to the other corners (from that corner) $\bf p_1,p_2$:

1. Calculate the matrix ${\bf T} = [{\bf p_1, p_2}]^{-1}$
2. Build matrix ${\bf M}=\left[\begin{array}{cc}{\bf T}&-{\bf T}d({\bf o}) \\ {\bf 0}& {\bf I}_2\end{array}\right]$
3. Build a vector $\bf v$ of the point you want to decide for.
4. Calculate $\bf Mv$
5. If
   1. any of the entries of $\bf Mv$ are $< 0$ OR
   2. both are positive and the sum $> 1$

Just to show it works: