Suppose i have estimated three points $\hat A$, $\hat B$ and $\hat C$ of a rectangle $ABCD$.
The fourth point can easily be: $\hat D=\hat A+(\hat C-\hat B)$.
But these points do not necessarily satisfy $\angle \hat A\hat B \hat C=90^\circ$ or the same for any other vertex. So $\hat A\hat B\hat C\hat D$ is not a rectangle.
Is there a proper way to correct those points in order $ABCD$ be a rectangle, under a minimal geometric error wrt the original points?.
We could assume the geometric error is: $$\sum_{X \in \{A,B,C,D\}} |X-\hat X|^2$$. Or any other metric.
I've finally set up this optimization problem with $x_i^0$ the original vertexs and $x_i$ the new vertex variable:
$$ \mathscr{P}) \min \sum_{i=1}^4 |x_i-x_i^0|^2\\ (x_1-x_2)^T(x_2-x_3)=0\\ x_1-x_2+x_3-x_4=\mathbb{0}\\ x_i^0 \in \mathbb{R}^2 $$
Which is solved in Matlab as:
with the square error:
And the rect angle constraint and fourth point constraint:
The result is depicted through: