I am working on an app that will transform a figure such as this:
Into this:
In short, the grey "canvas" is deformed so that the inner black quadrangle is as close to a rectangle as possible, while attempting to conserve the rectangles area if possible. Please assume that in the second image the black figure is much, much closer to a rectangle.
iOS already offers a method for deforming the canvas by stretching the corners to four new points. I just need to apply a generic solution for calculating the new points.
I am at a loss as to a consistent method; simply pulling a corner in a single direction is not consistent.
Do I need to rotate the corner through an arc? Do I need to move the new corner points diametrically away from the center of the grey figure?
All suggestions appreciated.
Thompson
After the discussion in the comments, I assume that the proportions of the desired rectangle are known to be $a:b$.
Though the method I describe below is good for understanding, it is not suitable for direct implementation. The resulting equation system is typically ill-conditioned. (It is also possible to get rid of the $\lambda_i$ by using cross products to express collinearity, and thus get a slightly smaller system to solve. This is described in detail in Multiple View Geometry by Hartley and Zisserman.)
Let $\boldsymbol{p}_i = (x_i, y_i, 1)$ be the homogeneous coordinates of the four corners in the input image. Let their desired destinations be $$ \left\{ \begin{aligned} \boldsymbol{q}_1 = (0, 0, 1) \\ \boldsymbol{q}_2 = (a, 0, 1) \\ \boldsymbol{q}_3 = (a, b, 1) \\ \boldsymbol{q}_4 = (0, b, 1) \end{aligned} \right.. $$ The corners are related through a planar homography, represented by a $3\times 3$-matrix $\boldsymbol{H}$, and the relations are $\lambda_i\boldsymbol{q}_i=\boldsymbol{H}\boldsymbol{p}_i$ for some scalars $\lambda_i$.
How does one determine $\boldsymbol{H}$? Let $\boldsymbol{h}_1^T$, $\boldsymbol{h}_2^T$ and $\boldsymbol{h}_3^T$ be the rows of $\boldsymbol{H}$. Then $$ \left\{ \begin{aligned} \lambda_i q_{i1} = \boldsymbol{h}_1^T\boldsymbol{p}_i \\ \lambda_i q_{i2} = \boldsymbol{h}_2^T\boldsymbol{p}_i \\ \lambda_i q_{i3} = \boldsymbol{h}_3^T\boldsymbol{p}_i \end{aligned} \right. \Longleftrightarrow \left\{ \begin{aligned} \boldsymbol{p}_i^T\boldsymbol{h}_1 - \lambda_i q_{i1} = 0 \\ \boldsymbol{p}_i^T\boldsymbol{h}_2 - \lambda_i q_{i2} = 0 \\ \boldsymbol{p}_i^T\boldsymbol{h}_3 - \lambda_i q_{i3} = 0 \end{aligned} \right.. $$ Write this system of equations in matrix form, $$\begin{bmatrix} \boldsymbol{p}_1^T & \boldsymbol{0} & \boldsymbol{0} & q_{11} & 0 & 0 & 0 \\ \boldsymbol{0} & \boldsymbol{p}_1^T & \boldsymbol{0} & q_{12} & 0 & 0 & 0 \\ \boldsymbol{0} & \boldsymbol{0} & \boldsymbol{p}_1^T & q_{13} & 0 & 0 & 0 \\ \boldsymbol{p}_2^T & \boldsymbol{0} & \boldsymbol{0} & 0 & q_{21} & 0 & 0 \\ \boldsymbol{0} & \boldsymbol{p}_2^T & \boldsymbol{0} & 0 & q_{22} & 0 & 0 \\ \boldsymbol{0} & \boldsymbol{0} & \boldsymbol{p}_2^T & 0 & q_{23} & 0 & 0 \\ \boldsymbol{p}_3^T & \boldsymbol{0} & \boldsymbol{0} & 0 & 0 & q_{31} & 0 \\ \boldsymbol{0} & \boldsymbol{p}_3^T & \boldsymbol{0} & 0 & 0 & q_{32} & 0 \\ \boldsymbol{0} & \boldsymbol{0} & \boldsymbol{p}_3^T & 0 & 0 & q_{33} & 0 \\ \boldsymbol{p}_4^T & \boldsymbol{0} & \boldsymbol{0} & 0 & 0 & 0 & q_{41} \\ \boldsymbol{0} & \boldsymbol{p}_4^T & \boldsymbol{0} & 0 & 0 & 0 & q_{42} \\ \boldsymbol{0} & \boldsymbol{0} & \boldsymbol{p}_4^T & 0 & 0 & 0 & q_{43} \\ \end{bmatrix} \begin{bmatrix} \boldsymbol{h}_1 \\ \boldsymbol{h}_2 \\ \boldsymbol{h}_3 \\ \lambda_1 \\ \lambda_2 \\ \lambda_3 \\ \lambda_4 \end{bmatrix} = \boldsymbol{0}, $$ and solve. Now, any point $\boldsymbol{p}$ in the input image is mapped to the point $\lambda\boldsymbol{q}=\boldsymbol{H}\boldsymbol{p}$. In particular, the corners are mapped to the corners of a rectangle.