I have the following optimization problem:
$\displaystyle \min_R \sum_{i=1}^n (X_i^T R Y_i)^2$
where $R \in \text{SO}(3)$, i.e. is a 3x3 rotation matrix, and $X_i,Y_i \in \mathbb{R}^3$.
If $n \le 6$, and I allow $R$ to be any 3x3 matrix, then I can compute a basis for $R$ from the null space of:
$\displaystyle A = \left[ \text{vec}(Y_1X_1^T), ..,\text{vec}(Y_nX_n^T)\right]$
so an equivalent problem is to find an $R \in \text{SO}(3)$ which is either in or closest to the null space of $A$. It seems that this might be a generalization of the orthogonal Procrustes problem.
Do any linear or eigen-decomposition-based solutions exist? If not, what methods have been used to efficiently solve this type of problem?