Let us consider vector bases $\mathcal{B_0} = (a,b,c)$, $\mathcal{B_1} = (i,j,k) $ and $\mathcal{B_2} = (x,y,z)$ in $\mathbb{R}^3$, such that :
- $\mathcal{B_1}$ is obtained by applying a rotation, defined by the matrix $R_1$, to $\mathcal{B_0}$
- $\mathcal{B_2}$ is obtained by applying a rotation, defined by the matrix $R_2$, to $\mathcal{B_1}$
So we have :
- $i = R_1 a$, $j = R_1 b$ and $k = R_1 c$
- $x = R_2 i = R_2 R_1 a$, $y = R_2 j = R_2 R_1 b$ and $z = R_2 k = R_2 R_1 c$
Thus, the two successive rotations can be directly described by just one matrix $R_3 = R_2 R_1$.
Now, introduce a vector $\mathbf{v}$, expressed as $\mathbf{v}_0$ in $\mathcal{B_0}$, $\mathbf{v}_1$ in $\mathcal{B_1}$ and $\mathbf{v}_2$ in $\mathcal{B_2}$.
By definition of a rotation matrix, the changes of basis from $\mathcal{B_0}$ to $\mathcal{B_1}$, and from $\mathcal{B_1}$ to $\mathcal{B_2}$ are expressed as :
- $v_0 = R_1 v_1$
- $v_1 = R_2 v_2$
So we have : $v_0 = R_1 R_2 v_2 = P_0^2 v_ 2$, where $P_0^2$ allows to change from basis $\mathcal{B_2}$ to basis $\mathcal{B_1}$.
So, if we take $v = x$, we have $x_2 = [1, 0, 0]^T$ and $x_0 = R_2 R_1 a_0$.
Since $a_0 = [1, 0, 0]^T = x_2$ $\implies$ $x_0 = R_2 R_1 x_2$ which is (obviously) different from $x_0 = R_1 R_2 x_2$
By definition of a rotation matrix, we should have $P_0^2 = R_3$, as for Euler angles rotation matrix. However the last line implies $P_0^2 \neq R_3$ as matrix product is not commutative. So where is the problem in my development ? Is it just that we do not compose change of basis like rotations ?
Let's look at a general problem: we have a basis $\mathcal{B}_0$ and a basis $\mathcal{B}_1$ and we want to get the coordinate matrix of the transformation $[\mathrm{Id}]_{1}^{0}$, so the transformation that takes coordinate vectors in $\mathcal{B}_0$ and takes it to $\mathcal{B}_1$. Let $x_i,y_i$ be the basis vectors of $\mathcal{B}_0,\mathcal{B}_1$ and suppose $P$ is a linear transformation such that $Px_i=y_i$. Let $[v]_{0/1}$ denote the coordinate vector of $v$ in $\mathcal{B}_{0/1}$ respectively.
Now let's define $[P]_0^0$ by saying that it takes $[v]_0$ to $[Pv]_0$ - this is achieved by setting the columns of $[P]_0^0$ to be $[Px_i]_0=[y_i]_0$. Therefore, this matrix takes $[y_i]_1$ to $[Px_i]_0=[y_i]_0$ so in fact $[P]_0^0=[\mathrm{Id}]_{0}^{1}=([\mathrm{Id}]_{1}^{0})^{-1}$.
In your case $P$ is a rotation. I think that basically the problem is that you never defined in which bases you're looking at $R_1$, $R_2$: are we talking $[R_1]^0_0$ or $[R_1]^1_1$ or $[R_1]_2^0$ or what? And the answer then depends. I presume you meant $[R_1]_0^0$ and $[R_2]_1^1$, in which case $$ [x]_0=[\mathrm{Id}]_{0}^{1}[\mathrm{Id}]_{1}^{2}[x]_2=[R_1]_0^0[R_2]_1^1[x]_2 $$ and also $$ [x]_2=[a]_0=[R_1^{-1}]_0^0[R_2^{-1}]_0^0[x]_0=([R_1]^0_0){-1}[\mathrm{Id}]_{0}^{1}[R_2^{-1}]_1^1[\mathrm{Id}]_{1}^{0}[x]_0=([R_1]^0_0)^{-1}[R_1]^0_0[R_2^{-1}]_1^1([R_1]^0_0)^{-1}[x]_0=([R_2]_1^1)^{-1}([R_1]^0_0)^{-1}[x]_0 $$ so again $$ [x]_0=[R_1]_0^0[R_2]_1^1[x]_2 $$ and there is no contradiction.
So, to recap: $[R_2]_1^1$ has as columns the coordinate expressions of $\mathcal{B}_2$ in $\mathcal{B}_1$ and $[R_1]_0^0$ has as columns the coordinate expressions of $\mathcal{B}_1$ in $\mathcal{B}_0$. Therefore, applying $[R_2]_1^1$ to $[x]_2$ gives the expression of $x$ in $\mathcal{B}_1$ and then applying $[R_1]_0^0$ to $[x]_1$ gives the expression of $x$ in $\mathcal{B}_0$
I think you approach this pretty much in anyway, though, as long as you keep a sharp distinction between a linear transformation and a matrix and make sure you always know what exact coordinate matrix your linear transformation has and in which basis you've decided to express it.
EDIT: eyeballfrog's answer says that the correct way is $v_0=R_2R_1v_2$. You also said in your question that obviously, $R_2R_1\neq R_1R_2$. So the answers we provided seem inconsistent. However, consider that both $$ [x]_0=[R_1]_0^0[R_2]_1^1[x]_2 $$ and $$ [x]_0=[R_2]_0^0[R_1]_0^0[x]_2 $$ In fact, $[R_1]_0^0[R_2]_1^1=[R_2]_0^0[R_1]_0^0$, so if you don't use coordinate transformations carefully, it looks like we would have $R_1R_2=R_2R_1$ - however, if we use the $[\cdot]$ notation, the confusion disappears as the $R$'s represent rotations in different bases. So in fact both answers are correct, it just amounts to different coordinate expressions.
EDIT: Also, what about defining $R_3=R_2R_1$, is everything still consistent? Well, since $R_3$ takes $\mathcal{B}_0$ to $\mathcal{B}_2$, we have $[R_3]_0^0=[\mathrm{Id}]_0^2=[\mathrm{Id}]_0^1[\mathrm{Id}]_1^2=[R_1]_0^0[R_2]_1^1$. It looks like $R_1$ and $R_2$ have flipped - but this is actually just what needs to happen. Indeed, we already know the first equality in the following line: $$ [R_1]_0^0[R_2]_1^1=[R_2]_0^0[R_1]_0^0=[R_2R_1]_0^0=[R_3]_0^0 $$ as required.