I want to change the basis from:
$$(|00\rangle,|01\rangle,|10\rangle,|11\rangle)$$ to $$(|00\rangle,|u_2\rangle,|u_3\rangle,|11\rangle)$$ ,
where $|u_2\rangle, |u_3\rangle = \frac{1}{\sqrt{2}}(|01\rangle\mp|10\rangle)$
The way I am doing it is by finding rotation matrix R and then using $A' = R.A.R^{-1}$.
I thought maybe I can rotate $|01\rangle,|10\rangle$ into $|u_2\rangle,|u_3\rangle $ respectively, while the other two map to themselves.
Thus I came up with the matrix: $$R=\begin{pmatrix}1&0&0&0\\0&\frac{1}{\sqrt 2}&\frac{1}{\sqrt 2}&0\\0&-\frac{1}{\sqrt 2}&\frac{1}{\sqrt 2}&0\\0&0&0&1\end{pmatrix}$$
The determinant checks out to be 1.
Is this method correct? Someone I'm getting wrong answers after finding the A' matrix using it. I am sparing the details of the later calculation here, because I suspect my error lies here itself.
From the description, it is much easier to write down the matrix for the reverse change of basis: that is, assuming that a vector $\mathbf v$ is given by $$\mathbf v = x_1 |00\rangle + x_2 |01\rangle + x_3|10\rangle + x_4|11\rangle = y_1 |00\rangle + y_2 |u_2\rangle + y_3 |u_3\rangle + y_4 |11\rangle,$$ it is much easier to find $(x_1,x_2,x_3,x_4)$ in terms of $(y_1,y_2,y_3,y_4)$.
Why? Because that information is basically given to us already. We have \begin{align} \mathbf v &= y_1 |00\rangle + y_2 |u_2\rangle + y_3 |u_3\rangle + y_4 |11\rangle \\ &= y_1 |00\rangle + y_2 \left(\frac{|01\rangle - |10\rangle}{\sqrt2}\right) + y_3 \left(\frac{|01\rangle + |10\rangle}{\sqrt2}\right) + y_4 |11\rangle \\ &= y_1 |00\rangle + \frac{y_2 + y_3}{\sqrt2} |01\rangle + \frac{-y_2 + y_3}{\sqrt2} |10\rangle + y_4 |11\rangle. \end{align} This tells us that $x_1 = y_1$, $x_2 = \frac{y_2 + y_3}{\sqrt2}$, $x_3 = \frac{-y_2 + y_3}{\sqrt2}$, and $x_4 = y_4$. In matrix form: $$\begin{bmatrix}x_1 \\ x_2 \\ x_3 \\ x_4\end{bmatrix} = \begin{bmatrix}1 & 0 & 0 & 0 \\ 0 & 1/\sqrt2 & 1/\sqrt2 & 0 \\ 0 & -1/\sqrt2 & 1/\sqrt2 & 0 \\ 0 & 0 & 0 & 1\end{bmatrix} \begin{bmatrix}y_1 \\ y_2 \\ y_3 \\ y_4\end{bmatrix}.$$
To find the change-of-basis-matrix going the other way, just take the inverse of this matrix. That's easier than it looks, because we just need the inverse of the middle $2\times 2$ block. Or, we can spot that the matrix is orthogonal, and therefore its inverse is its transpose. Either way:$$\begin{bmatrix}y_1 \\ y_2 \\ y_3 \\ y_4\end{bmatrix} = \begin{bmatrix}1 & 0 & 0 & 0 \\ 0 & 1/\sqrt2 & -1/\sqrt2 & 0 \\ 0 & 1/\sqrt2 & 1/\sqrt2 & 0 \\ 0 & 0 & 0 & 1\end{bmatrix} \begin{bmatrix}x_1 \\ x_2 \\ x_3 \\ x_4\end{bmatrix}.$$