Is there a general approach towards finding the intersection over union between two rotated cuboids? Or even a method that checks whether two rotated cuboids overlap?
At the moment, I sample points along each of the 12 edges of a cuboid and then check whether any of these points are inside the other cuboid. However, this is not an elegant solution and I was looking for something more mathematically straightforward. Is there any such method in math that can tell whether two rotated cuboids overlap?
Describe each cuboid using its center $\vec{c}$, orthonormal basis $\hat{e}_1$, $\hat{e}_2$, $\hat{e}_3$, where $$\hat{e}_i \cdot \hat{e}_j = \begin{cases} 1, & i = j \\ 0, & i \ne j \end{cases}$$ halved length/size $h_1$, $h_2$, and $h_3$ along each basis vector, and circumradius $r = \sqrt{h_1^2 + h_2^2 + h_3^2}$. This means the vertices are $$\begin{aligned} \vec{v}_0 &= \vec{c} + h_3 \hat{e}_3 + h_2 \hat{e}_2 + h_1 \hat{e}_1 \\ \vec{v}_1 &= \vec{c} + h_3 \hat{e}_3 + h_2 \hat{e}_2 - h_1 \hat{e}_1 \\ \vec{v}_2 &= \vec{c} + h_3 \hat{e}_3 - h_2 \hat{e}_2 + h_1 \hat{e}_1 \\ \vec{v}_3 &= \vec{c} + h_3 \hat{e}_3 - h_2 \hat{e}_2 - h_1 \hat{e}_1 \\ \vec{v}_4 &= \vec{c} - h_3 \hat{e}_3 + h_2 \hat{e}_2 + h_1 \hat{e}_1 \\ \vec{v}_5 &= \vec{c} - h_3 \hat{e}_3 + h_2 \hat{e}_2 - h_1 \hat{e}_1 \\ \vec{v}_6 &= \vec{c} - h_3 \hat{e}_3 - h_2 \hat{e}_2 + h_1 \hat{e}_1 \\ \vec{v}_7 &= \vec{c} - h_3 \hat{e}_3 - h_2 \hat{e}_2 - h_1 \hat{e}_1 \\ \end{aligned}$$ and its volume is $8 h_1 h_2 h_3$.
Then, if the centers are further away than the sum of the circumradiuses, $$\left\lVert \vec{c}_A - \vec{c}_B \right\rVert \gt r_A + r_B \tag{1a}\label{G1a}$$ which is easily calculated by squaring both sides, $$(\vec{c}_A - \vec{c}_B) \cdot (\vec{c}_A - \vec{c}_B) \gt (r_A + r_B)^2 \tag{1b}\label{G1b}$$ the cuboids cannot intersect (overlap) at all.
Otherwise, we can check if any of the vertices of one cuboid are inside the other cuboid. Vertex $\vec{v}$ is inside, if and only if $$-h_i \le \hat{n}_i \cdot ( \vec{v} - \vec{c} ) \le h_i \tag{2}\label{G2}$$ is true for all $i = 1, 2, 3$. If at least one vertex of one cuboid is inside the other cuboid, then the two do intersect.
Note that because of how the vertices are defined, and because dot product is distributive over vector addition, you only need to calculate 15 dot products (and not 24).
Otherwise, we need to examine if any edges of one cuboid intersects any face in the other cuboid. (We do need to test it both ways; it is not sufficient to test all twelve edges of one cuboid against the six faces of the other cuboid.)
Let the edge be between $\vec{w}$ and $\vec{w} + m \hat{\epsilon}$ (of cuboid B, the other vectors and $h_k$ referring to those of cuboid A). Calculate $$\begin{aligned} p_k & = \hat{e}_k \cdot (\vec{w} - \vec{c}) \\ q_k & = \hat{e}_k \cdot (\vec{w} - \vec{c} + m \hat{\epsilon}) \\ \end{aligned} \quad k = 1, 2, 3 \tag{3a}\label{G3a}$$ and then test the following points if they're contained in cuboid A, $$\begin{array}{lll} \vec{w} - \displaystyle m \frac{h_k + p_k}{q_k - p_k} \hat{\epsilon}, & \text{if} \quad p_k \lt -h_k \lt q_k \\ \vec{w} - \displaystyle m \frac{h_k + q_k}{p_k - q_k} \hat{\epsilon}, & \text{if} \quad q_k \lt -h_k \lt p_k \\ \vec{w} + \displaystyle m \frac{h_k - p_k}{q_k - p_k} \hat{\epsilon}, & \text{if} \quad p_k \lt h_k \lt q_k \\ \vec{w} + \displaystyle m \frac{h_k - q_k}{p_k - q_k} \hat{\epsilon}, & \text{if} \quad q_k \lt h_k \lt p_k \\ \end{array} \tag{3b}\label{G3b}$$ again for $k = 1, 2, 3$, but note that the points are on the faces of cuboid A (the one(s) perpendicular to $\hat{e}_k$), so you should ignore the face pair $i = k$ in the $\eqref{G2}$ test (since that is true by definition, as the above finds the point where the edge intersects with a face plane), and only test the other two. If true, the cuboids intersect.