Edit: There was a mistake in my OP. I removed the PSD constraints of each $M_i$ since $\lambda_i < 0$ i.e. they are negative-semidefinite. I apologize for this change. This in turn invalidates my proposed approach of rotating $y_i$ to the null space of each $M_i$.
Given
- $\{M_i\}_{i=1}^n$ as a set of $n$ 3x3 symmetric and negative-semidefinite matrices of rank 2 whose eigendecomposition is $\{Q_i\,D_i\,Q_i^\top\}_{i=1}^n$, with each $D_i = \text{diag}(\lambda_i, \lambda_i, 0)$ (i.e. two repeated eigenvalues $\lambda_i\in\mathbb{R}$ and one $0$-eigenvalue), and
- $\{y_i\}_{i=1}^n$ as a set of vectors, each $\in\mathbb{R}^3$.
I'm trying to find the rotation matrix, $R\in SO(3)$, that minimizes the function $f(R)$: $$ R = \arg\min_{R} f(R)= \arg\min_R \sum_i \left( y_i^\top R^\top M_i\, R\, y_i \right) = \arg\min_R \sum_i \left( y_i^\top R^\top Q_i\, D_i\, Q_i^\top R\, y_i \right). $$
(The following is wrong, see edit above) Since one eigenvalue of each $M_i$ is $0$, and, by letting $q_{0i}$ denote its associated eigenvector, I thought of simplifying the previous problem to: $$ R = \arg\max_{R} \sum_i \left( q_{0i}^\top R\, y_i \right), $$ Thus, aiming at the rotation that rotates the data vectors $y_i$ to the null space of each $M_i$. This new problem is similar to the Wahba's problem and hence can be solved in closed-form.
However (I may be ignoring some other details), I noted that this simplification suffers from the sign ambiguity of the eigenvectors $q_{0i}$. This is the reason why I currently think that this solution does not work in my original problem.
Given this observation, is there a way of choosing the right sign / orientation of each $q_{0i}$?
Any help is much appreciated.