Given a 3 by 2 rectangle I want to remove a $x$ by $y$ rectangle from a corner so that when I place the rectangle on the two new vertices it will "balance". By balance I mean the Center of Mass of the "truncated" rectangle $$(3-x)(2)(\frac{3-x}{2},\frac{2}{2})+(x)(2-y)(3-x+\frac{x}{2},y+\frac{2-y}{2})$$ (calculated using the additivity of Center of Mass as well as translation invariance. The formula for Center of mass of an $a$ by $b$ rectangle $(\frac{a}{2},\frac{b}{2})$.) will lie on the line which is perpendicular to the hypotenuse of triangle obtained by taking the diagonal of the missing $x$ by $y$ rectangle and also partition the triangle into two parts of equal area. I can't figure out how to calculate the point on the hypotenuse of the triangle with side lengths $x$ and $y$ which will perpendicular bisect the triangle and it seems to be complicated as there is a singularity where the parts you divide the triangle change from quadrilateral to triangle. My intuition tells me that the removed rectangle must be in a ratio $x$:$y$ equal to 2:3 but I want to logical validate this guess. I'm hoping these is a clever inner product/matrix way to solve this problem.
For the simpler case of a square, then $x$:$y$ are in ratio $1:1$ will work because the point on the hypotenuse which divides an isosceles triangle into equal parts is the midpoint of the hypotenuse. And the center of mass of the square can easily be calculated.
Suppose the rectangle has coordinates $(0, 0), (s, 0), (s, t), (0, t)$ and we truncate a rectangle with dimensions $a \times b$ at $(0, 0)$ so the truncated rectangle has coordinates $(0, 0), (a, 0), (a, b), (0, b)$. We can split this up into two rectangles along the $x = a$ line to compute the centre-of-mass. This gives the centre-of-mass of the truncated rectangle as \begin{align*} \frac{a(t-b)(\frac{a}{2}, \frac{t+b}{2})+t(s-a)(\frac{s+a}{2}, \frac{t}{2})}{st-ab} &= \left (\frac{s^2t-a^2b}{2(st-ab)}, \frac{st^2 - ab^2}{2(st - ab)}\right ) \end{align*} The line along which the centre-of-mass should lie is $2by - 2ax = b^2 - a^2$, so we have \begin{align*} 2b\left ( \frac{st^2 - ab^2}{2(st - ab)} \right ) - 2a\left ( \frac{s^2t-a^2b}{2(st-ab)}\right ) &= b^2 - a^2\\ b(st^2 - ab^2) - a(s^2t - a^2b) &= (b^2-a^2)(st-ab)\\ a(a - s) &= b(b - t) \end{align*} Note that this does not give us a conclusive value for $a : b$, but it gives us a relation that $a$ and $b$ should follow for the rectangle to balance. This also agrees with the result for a square.
In particular, for the $3 \times 2$ rectangle, we require $a(a - 3) = b(b - 2)$. Observe that the locus of values which fit this for $(a, b) \in [0, 3] \times [0, 2]$ are two hyperbolic arcs passing through the vertices of the this rectangle, as expected.
Edit: I realised I misread the post, and thought the line was a perpendicular bisector, not an area bisector.
The line along which the centre-of-mass should lie on the perpendicular to the hypotenuse which bisects the area of the triangle. If we assume $a \geq b$, then using similar triangles, the foot of this perpendicular on the hypotenuse is $\frac{a}{\sqrt{2}}$ away from $(a, 0)$, i.e. $\left (a-\frac{a^2}{\sqrt{2(a^2 + b^2)}}, \frac{ab}{\sqrt{2(a^2+b^2)}}\right )$, whch gives equation of perpendicular as $$by - ax = a\sqrt{\frac{a^2 + b^2}{2}} - a^2.$$ If instead $a \leq b$, then the foot of this perpendicular on the hypotenuse is $\frac{b}{\sqrt{2}}$ away from $(0, b)$, i.e. $\left (\frac{ab}{\sqrt{2(a^2 + b^2)}}, b - \frac{b^2}{\sqrt{2(a^2+b^2)}}\right )$, which gives equation of perpendicular as $$by - ax = b^2 - b\sqrt{\frac{a^2 + b^2}{2}}.$$ Substituting our centre-of-mass point in the $a \geq b$ case yields \begin{align*} b\left ( \frac{st^2 - ab^2}{2(st - ab)}\right ) - a\left ( \frac{s^2t - a^2b}{2(st - ab)}\right ) &= a\sqrt{\frac{a^2 + b^2}{2}} - a^2\\ \end{align*} Note that this expands out to give a quite disgusting expression which I'm not sure how to deal with.