Please help me in solving this problem, I am not sure what a transformation matrix R is and how to proceed.. Any help is appreciated.
Find the transformation matrix R that relates the (orthonormal ) standard basis of $\mathbb{C}^3$ to the orthonormal basis obtained from the following vectors via the Gram Schmidt process:
|a1> = $\begin{pmatrix} 1\\ i\\ 0 \end{pmatrix} $ |a2> = $\begin{pmatrix} 0\\ 1\\ -i \end{pmatrix} $ |a3> = $\begin{pmatrix} i\\ 0\\ -1 \end{pmatrix} $
They seem to be asking you to apply the Gram Schmidt process to those three vectors. Do it, that is what the exercise is about. Then (because those initial vectors are linearly independent) you will get three orthonormal vectors. Putting those three vectors as successive columns into a matrix will give a unitary matrix$~U$. Depending on the conventions used, your transformation matrix $R$ is either $U$ or its inverse (which is also $U^*$ since $U$ is unitary). In any case a linear operator given by $A$ on the standard basis will be given by $U^{-1}AU$ on the new (result of Gram Schmidt) basis.