I have a problem where I have N points with coordinates $X\in\mathbb{R}^{N\times 3}$. I have another point cloud with coordinates $\hat{X}\in\mathbb{R}^{N\times3}$. Now, I want to apply a translation to $\hat{X}$ (i.e., translate all the points equally) such that the sum of the Euclidean distances between the points in the two clouds is minimized.
I did a naive calculation in a single dimension. Then I want to minimize $$\sum_{i=1}^N (x_i - (\hat{x}_i - t))^2.$$
Taking the derivative with respect to t, setting it to 0 and solving for t gives me $$t = \frac{1}{N}\sum_{i=1}^N (\hat{x}_i - x_i).$$
I.e., the optimal translation is to remove the "average" difference between each point. However, I understood that I should take the periodic boundaries into consideration, so I changed $(\hat{x}_i - x_i)$ to the shortest vector between the points, taking the periodic boundaries into account.
However, I eventually found out that this sometimes fails, and (at least) one case it fails for is when the vectors between points in the two clouds are not "aligned" in the sense that they are cancelling each other out when computing the mean. See an example in 2D here. The different colors indicate the different point clouds. If I instead use this choice of vectors between points, which are not the shortest vectors, I obtain the correct minimal distance after translation.
Any idea how I can avoid such corner cases, so that I apply the translation so that the distance is actually minimized?