I am looking for an algorithm to check if a quadrilateral is convex. I am sure that the quadrilateral is not self-intersecting, so I only need to check if it is convex or concave. One method would be to check if any interior angle is larger than π. However, I am struggling to calculate the interior angle. Using the dot product just yields an angle between two lines, not necessarily the interior angle. Note that I intend to apply the algorithm to thousands of quadrilaterals programatically, so plotting is out of scope (and error prone to begin with).
Here is an example of a quadrilateral I would like to classify as convex or concave. The corners A, B, C and D are given in order. $A = \begin{pmatrix}2\\4\\-2\end{pmatrix}$, $B = \begin{pmatrix}-1\\3\\-1/7\end{pmatrix}$, $C = \begin{pmatrix}4\\2\\-12/7\end{pmatrix}$, $D = \begin{pmatrix}8\\1\\-20/7\end{pmatrix}$
EDIT
Let's start first with the assumption that the quadrilateral is planar and I will deal with the slight skwewedness later, maybe in a follow up question.
EDIT2
I just realized how ill defined my wish to include skew quadrilaterals was. Those are not even uniquely defined by 4 points.
Assuming you have the vertices in order, look at the cross products of the vectors of successive edges; with vertices $v_1 \dots v_4$ you'd look at $(v_3-v_2)\times (v_2-v_1)$ etc. These cross products will all point along the polygon's normal. For a convex polygon, they'll all point in the same direction.
You can simplify this more, knowing that the cross product $a \times a =0$.