I'm trying to simulate buoyancy forces on rectangular prism of arbitrary orientation. Suppose that the prism has size $L\times W\times H$ and an Euler angle of $(\phi, \theta, \psi)$. Furthermore, as a rudimentary approximation, the water can be represented as a waveless horizontal plane that slices the prism at some height $h$. A rough diagram is shown below, with the volume of interest highlighted in yellow:
Is there a way to calculate this volume in general?
Answer: $V = LW(d_y + H/2)$, where $d_y$ is the $y$-coordinate at which the surface of the water hits the $y$-axis.
If $\theta$ is the angle that the $y$-axis of the prism makes with the vector pointing straight up, then $d_y = h/\cos \theta$.
Note: This formula fails when the prism lies such that the $y$-axis is parallel to the surface of the water, in which case the $y$-axis intersection is non-unique. In this exceptional case, you could use one of the other axes to get the formulas $$ V = (d_z + L/2)WH, \qquad V = L(d_x + W/2)H $$ This formula also assumes that the entire "bottom" of the prism (relative to your chosen axis) is submerged.
Derivation:
I assume that your origin is taken to be in the center of the prism.
Here's an idea. Let $\mathbf n = (n_1,n_2,n_3)$ denote the normal vector to the surface water (relative to your intrinsic coordinate system of the prism). We ultimately won't need it for the final answer, but if you're interested this $\mathbf n$ can be calculated from your Euler angles as the second column of the rotation matrix corresponding to your Euler angles.
Let $d$ denote the $y$-coordinate at which the surface of the water hits the $y$-axis. The equation of the surface of the water has the form $$ n_1 x + n_2 (y - d) + n_3 z = 0 \implies \\ n_1 x + n_2 y + n_3 z = n_2 d \implies\\ y = d - \frac{n_1}{n_2}x - \frac {n_3}{n_2}z \implies\\ y = d - ax - bz $$ where $a = n_1/n_2$ and $b = n_3/n_2$. With that, your volume can be expressed as the triple integral $$ V = \int_{-W/2}^{W/2} \int_{-L/2}^{L/2} [(d - ax - bz) + H/2]\,dz\,dx\\ = \int_{-W/2}^{W/2} \int_{-L/2}^{L/2} (d+ H/2)\,dz\,dx = LW(d + H/2). $$ I suspect that this calculation can be avoided by some symmetry-argument.
In any case, the volume in question will be $LW(d + H/2)$, so all you need to do is find that $y$-coordinate $d$.