The equations $$ \begin{cases} x'=\frac{1}{\sqrt{2}}(x-z),\\ y'=\frac{1}{2}(x+y\sqrt{2}+z),\\ z'=\frac{1}{2}(x-y\sqrt{2}+z),\end{cases}\quad \begin{cases} x''=\frac{1}{\sqrt{2}}(y'-z'),\\ y''=-\frac{1}{2}(x'\sqrt{2}+y'+z'),\\ z''=\frac{1}{2}(-x'\sqrt{2}+y'+z'),\end{cases} $$ represent rotations of axes in three dimensions. Find the matrix of the resultant rotation and describe geometrically the net result of the two rotations.
Lets say this is the matrix we need: $$ \require{cancel} \cancel{\begin{pmatrix} x'''\\y'''\\z''' \end{pmatrix}= \begin{pmatrix} \frac{1}{2}& \frac{\sqrt{2}-1}{\sqrt{8}} & \frac{-2+\sqrt{2}} {4}\\ \frac{-2+\sqrt{2}}{4}& \frac{1}{4} & \frac{1-2\sqrt{2}}{4} \\ \frac{\sqrt{2}-1}{\sqrt{8}}& \frac{2\sqrt{2}+1}{4} & \frac{1}{4} \end{pmatrix}\cdot \begin{pmatrix} x\\y\\z \end{pmatrix}} $$ $\require{cancel}\cancel{\text{How to interpret, lets say, the rotation of $x$ axis?}}$
I basically forgot that matrix multiplication is not commutative. Correct matrix should be this:
$$ \begin{pmatrix} x'''\\y'''\\z''' \end{pmatrix}= \begin{pmatrix} 0&1&0\\-1&0&0\\0&0&1 \end{pmatrix}\cdot \begin{pmatrix} x\\y\\z \end{pmatrix} $$
A result - the $90 ^\circ$ rotation around $z$ axis.
That is not the right matrix. The right matrix is$$A=\begin{bmatrix}\frac12 & \frac{2-\sqrt{2}}4 & \frac{-2-\sqrt{2}}4 \\ \frac{-2-\sqrt{2}}4 & \frac14 & \frac{1-2\sqrt2}4 \\ \frac{2-\sqrt{2}}4 & \frac{1+2\sqrt2}4 & \frac14\end{bmatrix}.$$Its only real eigenvalue is $1$, its complex non-real eigenvalues are $\pm i$, and its eigenvectors are the multiples of $\left(\frac{\sqrt2}2,-\frac12,-\frac12\right)$. Therefore, it represents a rotation of $\frac\pi2$ radians around the line passing through the origin and $\left(\frac{\sqrt2}2,-\frac12,-\frac12\right)$. This can be checked as follows: if$$e_1=\left(\frac1{\sqrt2},-\frac12,-\frac12\right),\ e_2=\left(0,-\frac1{\sqrt2},\frac1{\sqrt2}\right),\text{ and }e_3=\left(-\frac1{\sqrt2},-\frac12,-\frac12\right),$$then $(e_1,e_2,e_3)$ is an orthonormal basis of $\mathbb R^3$, $A.e_1=e_1$, $A.e_2=e_3$, and $A.e_3=-e_2$.