I have a 3D euclidean space. Somewhere in this space is positioned unit cube, that is aligned with the axes of the 3D geometry. Assume that there is arbitrarily placed 2D square in the space that must not be aligned anyhow with the geometry, lets even say its convex polygon given by arbitrary points A, B, C, D.
Is there a fast algorithm to find the volume of the intersection of the pyramid 0, A, B, C, D with the cube?
Is there a chance to simplify the things if I know that A, B, C, D is rectangle relative to some (projective) plane, not necessary aligned with the axes? So that there exists camera matrix for camera centered at 0 such that I would like to find volume of the cut through the cube that gets projected on $[A, A+ \delta A]$ and $[B, B+ \delta B]$ for camera matrix/projective plane.
Here's a solution for your present issue (finding the polygonal shape of the cube face cut) :
We assume that the cube's vertices are
$$O(0,0,0), \ \ A(1,0,0), \ \ B(0,1,0), \ \ C(0,0,1), \cdots$$
For each vertex $V(x,y,z)$ of your polygon, consider the three intersection points $P_1,P_2,P_3$ of line $OV$ with the three planes with equations $x=1$, $y=1$, $z=1$ resp.
Then simply keep the one with smallest length $OP_k$.
Remark : The coordinates of intersection points are :
$$P_1(1,y/x,z/x), \ \ \ P_2(x/y,1,z/y), \ \ \ P_3(x/z,y/z,1) \tag{1}$$
assuming $x,y,z \neq 0.$
Therefore, finding the smallest length is in fact done by comparing
$$1/x, 1/y, 1/z$$
If $1/x$ is the smallest, then it is $P_1$, etc.
(Figure done with a Matlab program).