Background : I'm solving an engineering problem where I have to estimate the orientation of a body in 3D space. Usually, I use quaternions to do this, but I have to consider a special case where I know the true value of the yaw state. As such, I do not want to put this state into my filter. I have already solved this problem by using an Euler angle approach and various other ad-hoc hacks.
Question : Now I'm curious about the following question: What does the subspace of SO(3) or S(3) (where the quaternions live) look like corresponding to the space of Euler 321 angles where the yaw is always zero. In other words, if I limited the set of rotations strictly to a rotation about the body x axis, followed by a rotation about the body y axis, what would those rotations looks like in SO(3) (or any other parametrization of this space)?
I tried plotting the axis/angle combinations for all such rotations, and I can't find a pattern there. Neither can I find a pattern in the Quaterion space. Are you really forced to use all 4 parameters of the quaternions even when you limited yaw to be zero?
Sorry if this sounds too vague. I'm trying to find some leads to get me going.
I doubt you'll find an elegant answer, because as far as $SO(3)$ is concerned "yaw zero" is a relatively arbitrary limitation that refers to a relatively arbitrary decomposition of rotations; but here's something you can do.
(I find it easier to think in terms of rotations about fixed axes, so instead of your intrinsic rotation about $x$ followed by intrinsic rotation about $y$, I'll talk about the equivalent setup of extrinsic rotation about $y$ followed by extrinsic rotation about $x$.)
Note that your set includes exactly one non-trivial rotation about each rotation axis. The rotation axis is given by the fixed points of the rotation. If you want to fix a point under one of your rotations, you have to either not rotate at all, or if you do, there's exactly one point that the first rotation must take the desired fixed point to such that the second rotation can take it back. Thus, apart from the identity, you have exactly one rotation per axis, and you can characterize your set by specifying the angle as a function of the axis. This is not going to be a nice function, but you can calculate it as follows.
Your rotations are
$$ \pmatrix{1&&\\&\cos\phi_x&-\sin\phi_x\\&\sin\phi_x&\cos\phi_x}\pmatrix{\cos \phi_y&&-\sin\phi_y\\&1&\\\sin\phi_y&&\cos\phi_y}=\pmatrix{\cos\phi_y&\ldots&\ldots\\\ldots&\cos\phi_x&\ldots\\\ldots&\ldots&\cos\phi_x\cos\phi_y}\;. $$
Comparing this to the general rotation matrix for a rotation through $\theta$ about the unit axis $u$, which has $\cos\theta+u_i^2(1-\cos\theta)$ on the diagonal, gives you the rotation angle in terms of your two angles,
$$ 1+2\cos\theta=\cos\phi_x+\cos\phi_y+\cos\phi_x\cos\phi_y\;, $$
and a quadratic equation for the cosine of the rotation angle in terms of the rotation axis:
$$ \cos\theta+u_z^2(1-\cos\theta)=\left(\cos\theta+u_x^2(1-\cos\theta)\right)\left(\cos\theta+u_y^2(1-\cos\theta)\right)\;. $$
Fortunately you don't need to take a square root to solve this, since we know one solution, $\cos\theta=1$, the trivial rotation, so we can divide through by $1-\cos\theta$, leaving
$$ \cos\theta+u_z^2=u_x^2\cos\theta+u_y^2\cos\theta+u_x^2u_y^2(1-\cos\theta)\;, $$
or
$$ \cos\theta=\frac{u_x^2u_y^2-u_z^2}{(1-u_x^2)(1-u_y^2)}=\frac{u_x^2u_y^2+u_x^2+u_y^2-1}{u_x^2u_y^2-u_x^2-u_y^2+1}\;. $$
Not very nice, but I doubt that you're going to find something much better. Euler angles are ugly.
P.S.: Note the symmetry with respect to $x$ and $y$: You get the same set of rotations, independent of whether you turn first about $x$ and then about $y$ or vice versa.