Setup
We are given 2 triangles and a line such that one edge in each triangle is contained in that line. These 2 edges are such that they might overlap but their union forms a single continuous edge, i.e there are no "holes" between the 2 triangles.
Our goal is to modify the vertices of the triangles that are in the edge in such a way that the intersection at the line is empty and all points inside one triangle are closest to the final vertex of that triangle.
That's probably confusing so let's see a picture:
Given that setup, we seek to move the vertices $C,E,D,F$ into $C',E',D',F'$ such that $\overline{CD} \cup \overline{EF} = \overline{C'D'} \cup \overline{E'F'}$. With the added restriction that if $p \in \overline{C'D'}$ then $p$ is closer to $H$ than $K$ and if $p \in \overline{E'F'}$ then $p$ is closer to $k$ than $H$.
We can assume the triangles are coplanar, we can assume that they lie on the same side, we can assume no degenerate triangles. We cannot assume $H$ and $K$ are different nor that the other points do not overlap.
My "Solution"
It's in quotations because if this worked there would be no need to ask for help.
Given an arbitrary setup that obeys the above and breaking the rule of assuming the "origins" ($H$ and $K$) are not necessarily different. We can construct the perpendicular bisector to the line $\overline{HK}$ as in this picture:
Then for each of $E$ if $\vec{HE}\cdot \vec{LM} > 0$ (i.e it's on the left side of the bisector) we set $E'=M$. For $F$ we execute the same test (i.e if it's on the left of the line we set it to $M$).
We then do a similar test for $C$ and $D$ except this time we check if they are on the right side of the bisector.
If a triangle becomes degenerate after this process it's fully removed.
i.e if you get this:
You set $D' = F$ and entirely remove $EKF$ (which is intended since all points are now closer to $H$).
Problem
Aside the fact I am breaking an assumption to get this working, it seems this method has multiple blindspots, a randomly generated data set shows that problems occur in edge cases when for example 2 of the points overlap, the bisector is parallel to the line and a plethora of other tiny little things. I am wondering if someone can come up with a better solution.
For the sake of clarity, this is what the solution to the example would look like:
Explicit formulation:
Given a line segment $\overline{AB}$, you are given 2 triangles $CHD$ and $EKF$ such that $\{C,D, E, F\} \subset \overline{AB}$ (i.e the 4 points lie on the interior of the segment). And $\{H, K\} \not\subset \overline{AB}$.
We seek to find 4 new points ${C', D', E', F'}$ such that $|\overline{C'H}| < |\overline{C'L}|$; $|\overline{D'H}| < |\overline{D'L}|$; $|\overline{E'L}| < |\overline{E'H}|$; $|\overline{F'L}| < |\overline{F'H}|$.
That also satisfy $\overline{C'D'} \cup \overline{E'F'} = \overline{CD} \cup \overline{EF}$ and $\overline{C'D'} \cap \overline{E'F'} = \{p\}$ where $p \in \overline{AB}$, in other words, the 2 resulting segments overlap only at their boundary point and nowhere else.
It is possible that in certain cases that one of the 2 triangles cannot satisfy all of these restrictions(see the third figure). In such cases have the other triangle satisfy its conditions and mark the second triangle as "useless".