Lets take two vectors A and B represented in x,y,z(metres) and Euler angles XYZr(deg) as
A =
-0.0036 0.1235 -0.9213 120.2176 -0.4881 -87.1283
B =
-0.1241 -0.0112 -1.1028 -58.4000 1.6000 92.1000
Converting them to 4x4 Homogeneous matrix gives
M_A =
0.0501 0.9987 -0.0085 -0.0036
0.5023 -0.0326 -0.8641 0.1235
-0.8632 0.0390 -0.5033 -0.9213
0 0 0 1.0000
M_B =
-0.0366 -0.9989 0.0279 -0.1241
0.5245 0.0046 0.8514 -0.0112
-0.8506 0.0458 0.5238 -1.1028
0 0 0 1.0000
Transform B into A and A into B gives
M_BA = inv(M_B) * M_A
M_AB = inv(M_A) * M_B
M_BA =
0.9959 -0.0868 -0.0248 -0.0882
-0.0873 -0.9960 -0.0185 -0.1114
-0.0231 0.0206 -0.9995 0.2131
0 0 0 1.0000
M_AB =
0.9959 -0.0873 -0.0231 0.0830
-0.0868 -0.9960 0.0206 -0.1230
-0.0248 -0.0185 -0.9995 0.2088
0 0 0 1.0000
Converting M_BA and M_AB into Euler angles gives
angles_BA =
178.9396 -1.4222 4.9837
angles_AB =
-178.8198 -1.3244 5.0105
I was thinking that angles_BA and angles_AB should be same. But, they are not. Why there is a difference between angles_BA and angles_AB ??
You have two isometric transformations, $A$ and $B$, each of which is a combined translation and rotation.
Let's define some useful notation. Define $T_{X,Y,Z}$ as the matrix for a translation by $X$ units in the $x$ direction, $Y$ units in the $y$ direction, and $Z$ units in the $z$ direction. Define $R_{x;\theta}$ as the matrix for a rotation by the angle $\theta$ around the $x$ axis; likewise $R_{y;\theta}$ and $R_{z;\theta}$ for rotations around the $y$ and $z$ axes, respectively.
Also, let's write our angles in radians.
With a little help from WolframAlpha, I think I have deduced that $A$ is actually the transformation whose matrix is $$ M_A = T_{-0.0036,0.1235,-0.9213} R_{x;2.09819} R_{y;-0.008519} R_{z;-1.52068}. $$
That is, first you rotate by $-1.52068$ radians around the $z$ axis; then $-0.008519$ radians around the $y$ axis; then $2.09819$ around the $x$ axis; and finally you translate by the vector $(-0.0036,0.1235,-0.9213)^T.$ Note that the matrix on the right is the one whose transformation is applied first, because you apply the transformation by putting any coordinates you want transformed in a column vector on the right side of the matrix $M_A,$ and when you expand $M_A$ into a product of component transformations like this the rightmost matrix is closest to the column vector and is the first to be multiplied with it.
We could also say that $A$ is a rotation by $2.40815$ radians ($137.977$ degrees) around the axis through the origin in the direction of the vector $(0.674519, 0.638394, -0.370778)^T,$ followed by a translation by the vector $(-0.0036,0.1235,-0.9213)^T.$
Similarly, the matrix for $B$ is $$ M_B = T_{-0.1241,-0.0112,-1.1028} R_{x;-1.01927} R_{y;0.0279253} R_{z;1.60745}. $$
Now suppose we have a figure $F$ that is transformed by the rotation-translation $A$ into the figure $F_A$, whereas the rotation-translation $B$ transforms the figure $F$ into $F_B.$ Then the transformation that maps $F_B$ to $F_A$ is $B^{-1}$ followed by $A$, and the matrix of that transformation is $M_A M_B^{-1}$ (because, again, the matrix on the right is the matrix for the transformation that occurs first).
The matrix $M_B^{-1} M_A$ is something else altogether. I don't think it does what you think it does.
But since you are asking about a particular matrix you call $M_{BA} = M_B^{-1} M_A$, let's work with that matrix. This matrix works out to the following matrix product: $$ M_{BA} = T_{-0.0882,-0.1114,0.2131} R_{x;3.12309} R_{y;-0.0248221} R_{z;0.086982}. $$ Now you should expect that $M_{AB} = M_A^{-1} M_B$ is the inverse of $M_{BA},$ and indeed it is. You can confirm this by multiplying the two matrices together. But $$ M_{AB} = T_{0.0830,-0.1230,0.2088} R_{x;-3.12099} R_{y;-0.0231151} R_{z;0.0874497}. $$ As you observed, the angles do not match up in any obvious way.
The reason for mismatched angles is that $$ \left(R_{x;3.12309} R_{y;-0.0248221} R_{z;0.086982}\right)^{-1} \neq R_{x;3.12309}^{-1} R_{y;-0.0248221}^{-1} R_{z;0.086982}^{-1}. $$ That is, you don't reverse a sequence of rotations around the $x$ axis, then the $y$ axis, then the $z$ axis by making a sequence of equal (or opposite) rotations around the $x$ axis, then the $y$ axis, then the $z$ axis. If you want to re-use the angles, write $$ \left(R_{x;3.12309} R_{y;-0.0248221} R_{z;0.086982}\right)^{-1} = R_{z;0.086982}^{-1} R_{y;-0.0248221}^{-1} R_{x;3.12309}^{-1}. $$ That is, you can undo the sequence of rotations by exactly reversing the last rotation first, and then the second rotation, and then finally the first rotation. But if you insist on rotating around the $x$ axis first both times, both for $M_{BA}$ and also for the inverse transformation $M_{AB},$ you need different sets of angles. This will generally be true for all but the simplest cases. Neither matrix multiplication nor transformations in three-D space are commutative in general; in most cases, it matters in what order you do them.
Notice that the components of the translation also don't match; this is again because the transformations do not commute. In most cases, if you want to reverse a rotation-and-then-translation using the inverse of the original translation, you have to apply the inverse translation first.