I have a 3D triangular mesh and a cutting plane. I want to minimize the area of the cut intersection by rotating the normal of the plane.
The image shows the cross section of the cut and the mesh.
You can see the original mesh before the cut here.
Let's assume the cutting plane has a point A and a normal N. I want to change normal N in a way that the yellow area is minimum.
On
As cvanaret says, just cut perpendicular to the mesh, because it is basically a cylinder. Here is a proof of why a cut perpendicular to the cylinder yields the smallest cross section.
First, assume we cut the cylinder at an angle that is not perpendicular to the cylinder's length. This is what it should look like
The cross section is an ellipse. Since the ellipse does not change horizontally, its horizontal length $a$ is just equal to the radius $r$ of the cylinder. However, since we cut at an angle, the ellipses top is tilted back, and its bottom is tilted forward, so the vertical length $b$ is affected. From a side view, we get this picture.
The length $2l$ represents how much the cylinder's shape protrudes forwards due to the angled cut. From the Pythagorean theorem, we get that $ b = \sqrt{r^2+l^2}$. Also, $\sqrt{r^2+l^2} \gt \sqrt{r^2}$ because $l \gt 0$, meaning they can never be equal. Simplifying the previous inequality, we get $ b > r$.
Now, lets compare the cross sectional areas. If we cut it perpendicular to the cylinder's length, the area of the cross section is just the area of a circle with the cylinder's radius, or $\pi r^2$.
The cross sectional area of an angled cut cylinder is an ellipse with area $\pi ab$.
We will now prove that $\pi ab > \pi r^2$.
First off, we know that $b>r$. Now multiply both sides by $\pi r$ to get $\pi br > \pi r^2$. We know that $a=r$, so this inequality becomes $\pi ab > \pi r^2$. Thus, we have proven that the cross-sectional area of an angle-cut cylinder is greater than that of a perpendicular cut cylinder. That means to minimize the cross-sectional area, you need to cut perpendicular to the cylinder's length to get a circle cross-section instead of an ellipse one.
(Warning: I only have negligible experience in computational geometry. But I think this is an interesting engineering problem that worth more discussion. So I leave my incomplete idea below.)
Main idea
Consider the function $\phi:S^2\to \mathbb R$ that sends the unit normal of the cutting plane to the corresponding cross section area. Then $\phi$ is not differentiable only when the plane passing through a vertex of the mesh (*). (Not checked but seems likely.)
The plane already have a fixed point in your scenario. If the plane is also forced to contain a given vertex, then the unit normal would trace out a great circle on $S^2$. By (*), non-differentiable points are on the red curve in the following picture.
For each point in the mesh, draw the corresponding red curve. An intersection of two red curves reveals the normal direction when the plane passes through two vertice.
(For $S^2$) We are now looking for a local minimum of $\phi$ over the sphere (or hemisphere). IMO, we can first move along the red curves to find a minimal red-curve intersection point.
(For mesh) This corresponds to rotate a plane, from one vertex to another, while the other vertex fixed.
(For computation) Find out the four possible new vertice by finding out the smallest rotation angles below. Then compute the cross sectional areas, and select the one with smallest area.
Repeat the step to obtain a "local minimal cut through two points".
This may not be a true minimum. Take the long prism for example, the minimal cut does not pass through a vertex.
The hard thing is to find some methods to improve. Some possible ways are
subdivide some necessary facet and repeat the fore-mensioned method, or
descent into a nearby region,
,or 3. use the least square method to guess.
Let me explain (3) first. If we have a shape that resemble a tube or prism, then we fit the face normals with a plane, and use the direction of that plane. (Or should we just do this at the very beginning?)
Supppose we have a "local minimal cut through two points". Then, toward which direction should we descent or subdivide?
(For $S^2$) Now there are 8 directions for us to move in. But maybe it's not good to choose a vertex as the rotation pivot, so only 4 directions are remaining.
(For mesh) The 8 directions correspond to the following 8 ways to rotate the plane. So you can try some descent method in these direction.
One way is, in each direction we can find out the smallest angle $\theta$ to rotate the plane without passing through another vertex. Then rotate $\frac \theta 2$, and so on.
Or can we directly derive a formula for the cross sectional area of a prism? I gave up to derive that formula.
The unfinished computation
We write the plane as $ax+by=z$ (fixed point = origin) so that it may be easier to differentiate the area w.r.t $a,b$.
An edge $\overline{(p_1,p_2,p_3),(q_1,q_2,q_3)}$ of the mesh pass through the plane if $(p_3-ap_1-bp_2)(q_3-aq_1-bq_2)<0$. The the intersection (yellow dots) of the edge with the plane would be $$\frac{(p_3-ap_1-bp_2)(q_1,q_2,q_3)-(q_3-aq_1-bq_2)(p_1,p_2,p_3)}{(p_3-ap_1-bp_2)-(q_3-aq_1-bq_2)}$$
whose coordinates are $\frac{\text{linear function of a,b}}{\text{linear function of a,b}}$.
Find out these yellow dots and sort out the order (need some pre-processing about edges and faces) so that we could write down the cross products to calculate the area as $$\text{Area} = \frac 12 \left|v_1\times v_2 + v_2\times v_3+\cdots + v_n\times v_1\right|.$$
This is a quantity $$\textstyle\frac 12\sqrt{\left(\frac{\text{quadratic function of a,b}}{\text{quadratic function of a,b}}\right)^2+\left(\frac{\text{quadratic function of a,b}}{\text{quadratic function of a,b}}\right)^2+\left(\frac{\text{quadratic function of a,b}}{\text{quadratic function of a,b}}\right)^2}$$ depends on $a$. The coefficients are some constants involving $(p_1,p_2,p_3), (q_1, q_2, q_3), ...$ and $(r_1,r_2,r_3)$. But It seems too messy to differentiate.