I have a vector $v1$, and a rotated vector $v2$. I want to find two rotation matrices $Rx$ and $Ry$, which are rotation matrices around x-axis and y-axis, respectively, so that $Rx \times Ry \times v1 = v2$. My intuition tells me both $Rx$ and $Ry$ are unique if I restrict $\theta$, which corresponds to $Rx$ as between [-90, 90], and $\phi$, which corresponds to $Ry$ as between [0, 360]. But how am I going to find those matrices?
Make the problem simplified, let $v1=(0,0,1)$. I finally get this equation:
$\begin{pmatrix} \sin(\phi) \\ -\sin(\theta)cos(\phi) \\ \cos(\theta)cos(\phi) \\ \end{pmatrix}=\begin{pmatrix}x\\y\\z\end{pmatrix}.$
But I don't feel right about that. $\phi$ is determined by $x$? It's so easy to find a counterexample, e.g. $v1=(0, 0, 1), v2=(x, y, 0)$. In this case, $\phi$ should be 90 or 270. Where am I wrong?


The rotation about the $x$ axis by an angle $\theta$ is given by the rotation matrix:
$ R_x = \begin{bmatrix} 1 && 0 && 0 \\ 0 && c_1 && -s_1 \\ 0 && s_1 && c_1 \end{bmatrix} $
where $c_1 = \cos \theta, s_1 = \sin \theta $
And the rotation about the $y$ axis by an angle $\phi$ is given by the rotation matrix:
$R_y = \begin{bmatrix} c_2 && 0 && s_2 \\ 0 && 1 && 0 \\ -s_2 && 0 && c_2 \end{bmatrix} $
where $c_2 = \cos(\phi), s_2 = \sin(\phi) $
So, the combined rotation is
$ R = R_x R_y = \begin{bmatrix} 1 && 0 && 0 \\ 0 && c_1 && -s_1 \\ 0 && s_1 && c_1 \end{bmatrix} \begin{bmatrix} c_2 && 0 && s_2 \\ 0 && 1 && 0 \\ -s_2 && 0 && c_2 \end{bmatrix} = \begin{bmatrix} c_2 && 0 && s_2 \\ s_1 s_2 && c_1 && -s_1 c_2 \\ - c_1 s_2 && s_1 && c_1 c_2 \end{bmatrix} $
Now given two arbitrary unit vectors given as follows:
$v_1 = ( x_1, y_1, z_1 ) $ and $v_2 = (x_2, y_2, z_2 ) $
where $x_1^2 + y_1^2 + z_1^2 = 1 = x_2^2 + y_2^2 + z_2^2 $
And you want $v_2 = R v_1 $, so
$ \begin{pmatrix} x_2 \\ y_2 \\ z_2 \end{pmatrix} = \begin{bmatrix} c_2 && 0 && s_2 \\ s_1 s_2 && c_1 && -s_1 c_2 \\ - c_1 s_2 && s_1 && c_1 c_2 \end{bmatrix} \begin{pmatrix} x_1 \\ y_1 \\ z_1 \end{pmatrix} $
From the first component of the left and right sides of this vector equation we have
$ x_2 = c_2 x_1 + s_2 z_1 $
And this equation can be solved for $\phi$ as long as
$ | x_2 | \le \sqrt{ x_1^2 + z_1^2} $
Otherwise there is no solution.
Assuming this inequality is satisfied, then we have $\phi$ and therefore, $c_2$ and $s_2$.
The second component says
$ y_2 = s_1 s_2 x_1 + c_1 y_1 - s_1 c_2 z_1 = s_1 (s_2 x_1 - c_2 z_1) + c_1 y_1 $
And this equation will have a solution if and only if
$ | y_2 | \le \sqrt{ (s_2 x_1 - c_2 z_1)^2 + y_1^2 } $
And finally, from the third component, we have
$ z_2 = c_1 ( c_2 z_1 - s_2 x_1 ) + s_1 y_1 $
And it will have a solution if and only if
$ | z_2 | \le \sqrt{ (c_2 z_1 - s_2 x_1)^2 + y_1^2 } $
It can shown that the last two inequalities follow from the first one. So only one check is necessary, namely, that $|x_2| \le \sqrt{x_1^2+z_1^2} $.
If this inequality is satisfied, then we readily obtaine $\phi$ of $R_y$. And then to find $\theta$ of $R_x$, solve for $c_1$ and $s_1$ from the second and third components. These two equations can be written as the following linear system in $c_1 $ and $s_1$
$\begin{bmatrix} y_1 && s_2 x_1 - c_2 z_1 \\ c_2 z_1 - s_2 x_1 && y_1 \end{bmatrix} \begin{bmatrix} c_1 \\ s_1 \end{bmatrix} = \begin{bmatrix} y_2 \\ z_2 \end{bmatrix} $
From which
$ \begin{bmatrix} c_1 \\ s_1 \end{bmatrix} = \dfrac{1}{y_1^2 + ( s_2 x_1 - c_2 z_1)^2 } \begin{bmatrix} y_1 && c_2 z_1 - s_2 x_1 \\ s_2 x_1 - c_2 z_1 && y_1 \end{bmatrix} \begin{bmatrix} y_2 \\ z_2 \end{bmatrix}$
And this simplifies to
$ \begin{bmatrix} c_1 \\ s_1 \end{bmatrix} = \dfrac{1}{y_1^2 + (s_2 x_1 - c_2 z_1)^2 } \begin{bmatrix} y_1 y_2 + z_2 (c_2 z_1 - s_2 x_1) \\ (s_2 x_1 - c_2 z_1) y_2 + y_1 z_2 \end{bmatrix} $
Once we have the numerical value of $c_1$ and $s_1$, angle $\theta$ can be determined uniquely as
$ \theta = \text{atan2}(c_1, s_1) $
So, in summary, if there are solutions, we can have a single solution (this case occurs when $| x_2 | = \sqrt{ x_1^2 + z_1^2}$ . Otherwise, there will be exactly two solutions for $\phi$ and a corresponding unique $\theta$ for each of the two $\phi$'s.