which of the following are the rotation matrices?
I know that if det|matrix|=1 or M.Mt=I=Mt.M is the property that makes some matrix a rotation matrix. but in this case options (1,3,4) seems to be alright as there determinant is 1 and above properties hold. but my option is not getting accepted.
On
Number $4$ is not a rotation matrix. If $\ X=M^\top M\ $ then $$ X_{11}=0.3835^2+0.5710^2+1.3954^2\approx 2.42\ne1\ \text{, and}\\ X_{21}=0.5710\times0.3835+0.5919\times0.5710-0.0217\times1.3954\approx0.527\ne0\ . $$ Number $1$ is also not exactly a rotation matrix because $\ X=M^\top M\ $ is not exactly the identity in that case. It is sufficiently close, however, that the entries of $\ M\ $ could be regarded as being approximations, correct to $5$ significant figures, of those of a rotation matrix.
I'd suggest, therefore, trying the answer $1$, $2$ and $3$, and then, if that doesn't work, just $2$ and $3$.
The correct options are (2) and (3).
The problem with options (1) and (4) are that they dont have determinant equal to 1.
(1) has determinant 1.00007595810000
(4) has determinant 1.00006320846200
I guess you did the mistake of assuming these determinant values as 1 since they are "close enough" to 1.
But, in the definition of rotation matrices, there is no "close enough" term. If it says determinant of a rotation matrix has to be 1, it has to be 1. See options (2) and (3) for example. They have determinants 1.