Is there a general method to convert a matrix transformation from one coordinate system to another, so that the resulting transformation looks the same on screen?
For example, there are some transformations in a coordinate system with X right, Y up, and Z toward the viewer. And they need to be converted to a coordinate system with X right, Y away from the viewer, and Z up.
What would be the operation that needs to be performed for each matrix so that the transformations look the same in the other coordinate system? And is there a general way to construct this operation given the source and destination basis vectors?
Yes, the linear transformation can be transformed from one frame to the other one.
Supposed you have transformation $y=Ax$ in the frame $\{0\}$.
This transformation in the new frame $\{1\}$ looks like $Ry=R(A(R^{-1}R)x)= (RAR^{-1})(Rx)$, hence the transformation in the new frame has form $RAR^{-1}$, where $R$ is transformation from $\{0\}$ frame to the frame $\{1\}$.
If this transformation is orthogonal we have $RAR^T$.