Find out if a triangle is inside a rectangle

Question

So, this question came to my mind, but I didn't find any answer for it. What I could do to check if a triangle is inside a rectangle, if the points of the rectangle are given (A,B,C,D), and the points of the triangle too (a,b,c), where: $$A=(X_1,Y_1)\\ B=(X_2,Y_2)\\ C=(X_3,Y_3)\\ D=(X_4,Y_4) \\a=(x_1,y_1)\\ b=(x_2,y_2)\\ c=(x_3,y_3)$$

Answer 1

Here's an approach based on vector arithmetic that doesn't require you to figure out how to rotate the rectangle parallel to the coordinate axes.

This approach does require you to identify two vertices that are diagonally opposite each other. If the vertices were given in either clockwise or counterclockwise order, this is easy: $A$ and $C$ are diagonally opposite. For the following I'll assume the vertices are labeled so that indeed $C$ is diagonally opposite $A$.

Let \begin{align} u_1 &= X_2 - X_1, \\ v_1 &= Y_2 - Y_1, \\ u_2 &= X_4 - X_1, \\ v_2 &= Y_4 - Y_1. \end{align}

Now compute the following four quantities: \begin{align} a_1 &= X_1 u_1 + Y_1 v_1 , \\ b_1 &= X_2 u_1 + Y_2 v_1 , \\ a_2 &= X_1 u_2 + Y_1 v_2 , \\ b_2 &= X_4 u_2 + Y_4 v_2 . \end{align}

Now to tell if any point $(x_k, y_k)$ is in the rectangle, calculate \begin{align} p_1 &= x_k u_1 + y_k v_1 , \\ p_2 &= x_k u_2 + y_k v_2 . \end{align} The point $(x_k, y_k)$ is inside the rectangle if and only if $a_1 < p_1 < b_1$ and $a_2 < p_2 < b_2$.

If you consider points on the perimeter of the rectangle to be "inside," just write $\leq$ instead of $<$ in the preceding inequalities.

Note: This works only for rectangles, not for other kinds of quadrilaterals.

---

For anyone wondering why this works:

This method works by using properties of the vector inner product ("dot product") and by exploiting the fact that the sides of the rectangle are perpendicular to their adjacent sides.

Treated as vectors, $(u_1,v_1)$ is $B - A$, which is a vector perpendicular to side $AD$, and $(u_2,v_2)$ is $D - A$, which is a vector perpendicular to side $AB$. For a point $P$ at coordinates $(x,y)$, viewed a a vector from the origin, $$ x u_1 + y v_1 = P \cdot (B - A). $$ For varying choices of $P$, the product $P \cdot (B - A)$ is proportional to the distance to $P$ from a line $\ell$ through the origin perpendicular to $B - A$, measured in the direction from $A$ to $B$. Now we invoke the fact that $ABCD$ is a rectangle: since $\ell$ and the line $AD$ are both perpendicular to $AB$, they are parallel to each other; therefore all points on the line $AB$ are on the same side of $\ell$ and all are at the same distance from $\ell$. If $P \cdot (B - A) = A \cdot (B - A)$ then $P$ is on the same side of $\ell$ as $A$ and at the same distance, therefore $P$ is on the line $AD$. But if $P \cdot (B - A) > A \cdot (B - A)$ then $P$ is on the same side of $AD$ as $B$ is. Similarly, if $P \cdot (B - A) = B \cdot (B - A)$ then $P$ is on the same side of $\ell$ as $B$ and at the same distance, therefore $P$ is on the line $BC$, but if $P \cdot (B - A) < B \cdot (B - A)$ then $P$ is on the same side of $BC$ as $A$ is.

To test whether $P=(x_k,y_k)$ is inside the rectangle, therefore, we can compute the inner products as in the first part of this answer, then test the four inequalities $a_1 < p_1$, $p_1 < b_1$, $a_2 < p_2$, and $p_2 < b_2$. If all four inequalities are true, this confirms that $P$ is on the same side of $AD$ as $B$ is, on the same side of $BC$ as $A$ is, on the same side of $AB$ as $D$ is, and on the same side of $CD$ as $A$ is, in which case $P$ is inside the rectangle. If any of the inequalities is false then $P$ is on one of the sides or on the "wrong" side of one of the lines that lie along the sides and is outside the rectangle.

Answer 2

It is clear that you can take WLG the rectangle $$A=(X_1,Y_1)\\ B=(X_2,Y_2)\\ C=(X_3,Y_3)\\ D=(X_4,Y_4)$$ with parallel sides to the axes. It follows the region enclosed by the rectangle is defined by the product $$[X_1,X_2]\space\text {x}\space [Y_1,Y_4]=\{(x,y)\space |\space X_1\le x\le X_2;\space Y_1\le y\le Y_4\}$$ Thus $$a=(x_1,y_1)\\ b=(x_2,y_2)\\ c=(x_3,y_3)$$ being the triangle, one has

$$\text{ The triangle is inside the rectangle }\iff (x_i,y_i)\in[X_1,X_2]\space\text {x}\space [Y_1,Y_4];\space i=1,2,3 $$

Answer 3

Draw three horizontal rays from the vertex of the triangle.