I have two sets of 3-D vectors $x_i$ and $y_j$, of say $N$ and $M$ elements, respectively. I want to compute a rotation matrix whose X axis points towards $x_i$, and its $Z$ axis points towards $(x\times y)$, or in other words, $y_i$ lay on the $XY$ plane.
If $N=M=1$, the rotation matrix can be found by fixing $x$, so $z = x \times y$ and $\hat{y} = x \times z = (x \times y) \times x$.
Then, the rotation matrix is $R = [x, \hat{y}, z]$.
However, in my application, $x$ and $y$ vectors are noisy readings from sensors, so instead I have two sets of vectors, that is, $N>>1$ and $M>>1$.
How can the optimal rotation matrix (or quaternion) be found from such a dataset? The approach I am taking now is to use the mean of both sets to reduce to the default method, but there should be an approximation method to solve that.