Given $4$ arbitrary points in the space $A(x_1,y_1), B(x_2,y_2), C(x_3,y_3,), D(x_4,y_4)$, how do you check if an arbitrary point $X(x_5,y_5)$, is within the quadratic area marked by the $4$ points $A,B,C$ and $D$?
Edit:
I apologize for the mistake previously. Yes I meant quadratic area determined by 4 complanar points
Assuming the four points are in the same plan (coplanar).
There are six lines defined by the 4 points in the plan.
Write the six line equations as {aX + bY = 0} - the 4 lines that define the quadratic area will have the property that for each such line the other two points are on the same side of the line on the plan (calculate the value of aX+bY - to be on same side the sign of the values has to be the same).
After selecting the 4 right lines - the area of the quadrilateral is well defined - any point in the area will have result the same signs as derived in the process of selecting the four quadrilateral lines (the signs derived for the arbitrary point need to have the same sign as the two points out of the line, for the four lines).
All the points that are not matching the sign - are outside the quadrilateral area.