I have a standard orthonormal coordinate frame $[x,y,z]$ in Euclidean space. In its initial state it is arbitrarily rotated.
I would like its last vector $z$ to be aligned upwards $Y = [0, 1, 0]$ as much as possible, i.e. trying to minimize their angle. There is, however only a single degree of rotational freedom, I can only rotate the frame around its $y$ axis.
Of course, when $|y \cdot Y| = 0$ (i.e. $|z \cdot Y| = 1$) there is no solution, but how to compute the oriented angle $r_y$ to achieve the best alignment otherwise?
Update
Perhaps this could lead to the solution. Since only rotation around $y$ is possible, that vector must stay the same, but the remaining two will rotate. Then the angle between $z$ and $Y$ could be minimal when $x$ is perpendicular to $Y$,
- $x' = \lVert y \times Y \rVert$
- $z'= \lVert x' \times y \rVert$
I suspect the solution could be the frame $[x', y, z']$ but I can't prove it so far.
Let $R_0 = [x , y, z ] $ , and you want to rotate this frame about the $y$ axis, then the resulting frame is
$R' = [x', y', z'] = R_0 R_y $ where
$ R_y = \begin{bmatrix} \cos \theta && 0 && \sin \theta \\ 0 && 1 && 0 \\ - \sin \theta && 0 && \cos \theta \end{bmatrix} $
Hence,
$ z' = x \sin \theta + z \cos \theta $
Since we want $z'$ to be aligned with $Y = e_2 = [0, 1, 0]^T$ as much as possible, then we want the angle $\phi$ between them to be as small as possible. We have,
$\cos \phi = z' \cdot Y = x_2 \sin \theta + z_2 \cos \theta $
We're seeking the maximum of $\cos \phi$ as it corresponds to the minimum $\phi$.
Combining the two sinusoids in the above expression
$ \cos \phi = \sqrt{ x_2^2 + z_2^2 } \cos( \theta - \psi ) $
where $\psi = \operatorname{atan2}( z_2, x_2 ) $ , i.e. $\psi$ is the only solution to $\cos \psi = z_2 $ and $\sin \psi = x_2 $.
The maximum of the above shifted $\cos$ function occurs when $\theta = \psi$. This also implies that the minimum angle possible between the rotate $z$ axis and the $Y$ axis is
$ \phi_{Min} = \cos^{-1} \sqrt{x_2^2 + z_2^2} $
Note:
The rotated frame is given by $R = R_0 R_y$ and not $R_y R_0$ because we are rotating the frame about one of its local axes. To prove this, consider a point $p= (u,v,w)$ whose coordinates are given in world coordinates, i.e
$ p = u e_1 + v e_2 + w e_3 $
Its coordinates with respect to the given frame $R_0$ are given by
$ q = R_0^T p $
This means that
$ p = R_0 q $
Remember that $q$ is the coordinates of the given point as expressed with respect to the basis $R_0$. If we now rotate this frame with respect to its original orientation about its $y$ axis, then we can write
$ q = R_y r $
where $r$ is the coordinate vector of the same point but when expressed with respect to the rotated $R_0$. Combining the above, we have
$ p = R_0 q = (R_0 R_y) r $
It follows that the expression for the rotated frame is $R_0 R_y$.