How to extract the angle a from the rotation matrix, given by:
rotation=np.matrix([[1, 0, 0, 0],
[0, math.cos(4*math.radians(a)), math.sin(4*math.radians(a)), 0],
[0, math.sin(4 * math.radians(a)), -math.cos(4 * math.radians(a)), 0],
[0, 0, 0, -1]])
a=60
X=np.matrix('1 ;-1 ;0 ;0')
a1=rotation*X
a1=[[ 1. ]
[ 0.5 ]
[ 0.8660254]
[ 0. ]]
In the case if I have the value of a1 and the rotation, how to get back the value of a?
Thank you.
It actually depends on what your 4D space is - what subset of rotations are allowed.
If this is a true 4D space, then rotations require two angles (you can't write every rotation as a pure rotation about some axis).
However, consider now a case where one of the coordinates is unchanged (one of the lines of the matrix has 1 on the diagonal and 0 elsewhere). In that case, the rest of the matrix (call it A) is 3×3, and can be decomposed as such:
$$\operatorname{Tr} A=2\cos\phi+1$$ $$A+A^T=\begin{bmatrix}0& v_z & -v_y \\-v_z & 0 & v_x\\v_y&-v_x & 0\end{bmatrix}$$ where $(v_x,v_y,v_z)$ is the axis of rotation (it's not normalized, the length is $\sin\phi$ which also tells you the angle).
You must really be careful, though. Your given matrix is not even a rotation. Its determinant is not equal to $1$. This is why you need to be sure what kind of transformations are all that are possible in your case, and find what the invariants in your case are. Trace and determinant are always good things to check, because they don't depend on the choice of the coordinate basis. But if your matrix is always aligned in some special form (for example, always in the form you stated above), then you can simply look at the components and invert the functions (arcsin of a component or something like that).
Moreover: as you mentioned this matrix occurs in "rotating" the half wave plate. This doesn't sound like you have a rotation matrix, but a matrix composed of rotation and phase retardation, something like $R^TYR$ where $R$ is rotation and $Y$ something diagonal that inverts the phase. Is that correct? Because in that case, your question is misleading, you're then actually asking about what rotation you need to apply to a diagonal matrix to get a desired response, that's a different question. In any case, it would be helpful to know what your basis is - what are the 4 components you are acting on. Is this Jones formalism?
Edit:
You specified that these are Mueller matrices, and the vectors must then be Stokes vectors. This is not true 4D space. The first component is intensity and is untouched by these operations. The rest of the vector behaves as a 3D vector of polarization, bot that does not map to 3D space, but is a combination of a 2D rotation (polarization vector) and a complex rotation (phase between components).
As a result, components don't behave the same way, and you can't really have a general rotation, just 2D rotations in components 2 and 3, as you specified. The Mueller matrix is also not a rotation matrix but a rotated matrix of polarization reversal. It can rotate linear polarization and flip circular polarization, but it can't do a general transformation between circular and linear polarizations. For that, you need other optical elements.
Now for the solution. I now understand you want to know what matrix you need to take $(1,-1,0,0)$ and turn it into "vector" ${\bf a}$.
For most values of the vector, there is no solution. If first component is not equal to 1, waveplate is not enough and you need also attenuation (absorption). If last component is not equal to 0, you need something that creates circular polarization out of linear (for example, a quarter wave-plate). Only if it stays in the linear polarization and also keeps normalization (norm of the 2D vector made of components 2 and 3 stays normalized), then you simply realize from the second column of the matrix that $$(1,-1,0,0)\mapsto (1,-\cos 4\alpha,-\sin 4\alpha,0)$$ This means that you can reconstruct $\tan 4\alpha={\bf a}_3/{\bf a}_2$.
If you want to create a general state with an arbitrary vector, you have to know what optical elements (and what matrices) you have available to multiply together. Then, you can start reconstructing what you need. And in that case, understanding of your underlying physical case is more important than brute force algebra.