I have set of 2D affine transformations of images and I need to modify the transformations such way that they become as close to rotations as possible to minimize distortions of images.
Let the transformations be defined by 2x2 matrices $A_i$ and the problem is to find a single matrix $H$, such that all $H A_i$ are nearly orthogonal.
I'm not sure how to measure orthogonality, but I tried to use the following norm: $\|H A_i(H A_i)^T-I\|$. However it expands to fourth degree polynomial of 4 variables (coefficients of $H$). Minimization of sum of these expressions reduces to a set of third degree equations and I don't know how to solve it.
Is there any other approach to solve the problem?
I have found a solution that works rather well for me.
To compute the cost for a single matrix I used singular value decomposition. Let $\sigma_{1,i}$ and $\sigma_{2,i}$ will be the largest and the smallest singular values of 2x2 matrix $A_i$ accordingly. Then I defined the cost as pair $c_i=({\sigma_{1,i}\over\sigma_{2,i}} - 1, \sigma_1\sigma_2-1)$. The first term penalizes stretching and the second one — scaling of the transformed image.
The overall cost for the set of matrices is sum of squares of individual costs: $C = \sum_i{c_i^2}=\sum_i({\sigma_{1,i}\over\sigma_{2,i}}-1)^2+(\sigma_1\sigma_2-1)^2$.
Then I used Levenberg-Marquardt algorithm to minimize this cost. The variables are coefficients of the matrix $H$.
However, I've implemented more general version of this algorithm that works for homographies, but not only for affine transformations. So, it removes perspective distortions as well.