It is given that $f: \mathbb{R}^2 \to \mathbb{R}^3$ $$f(x, y) = (2x-3y,\ x,\ 5x+y)$$
The bases are given by as follow:
- $P = \{(1, -1), (4, 0)\}$
- $Q = \{(1, 1, 0), (1, 0, 1), (0, 1, 1)\}$
My approach:
The transformation matrix w.r.t standard (canonical) basis is $A$, $$A = \begin{bmatrix} 2 & -3\\ 1 & 0\\ 5 & 1 \end{bmatrix}$$
Change of basis matrix wrt $P$ basis is $C_1$, such that $C_1 \vec {x_P} = \vec x$
$$C_1 = {1 \over 4}\begin{bmatrix} 0 & -4\\ 1 & 1 \end{bmatrix} \text{ and } C_1^{-1} = \begin{bmatrix} 1 & 4\\ -1 & 0 \end{bmatrix} $$
Change of basis matrix wrt $Q$ basis is $C_2$, such that $C_2 \vec {y_Q} = \vec y$
$$C_2 = {1 \over 2}\begin{bmatrix} 1 & 1 & -1\\ 1 & -1 & 1\\ -1 & 1 & 1 \end{bmatrix} \text{ and } C_2^{-1} = \begin{bmatrix} 1 & 1 & 0\\ 1 & 0 & 1\\ 0 & 1 & 1 \end{bmatrix} $$
Now, $$\vec y = A \vec x$$ $$C_2 \vec {y_Q} = A C_1 \vec {x_p}$$ $$\vec {y_Q} = C_2^{-1} A C_1 \vec {x_p}$$ $$\vec {y_Q} = D\ \vec{x_p}$$ Now D is given by $$D = C_2^{-1}AC_1$$
What is the problem in this method? I am not able to get the correct answer.
It seems you're confusing the change of basis matrix and its inverse. The columns of a change of basis matrix $P$ are made up of the coordinates of the vectors in the new basis in the old basis, and the relation between the coordinates $X$ of a vector in the old basis, $X'$ its coordinates in the new basis (seen as a single-column matrix) is $$X=PX'.$$
Therefore, here, if I denote $P$ the change of basis matrix in $\mathbf R^2$, $Q$ the change of basis matrix in $\mathbf R^3$, we have $Y=AX$, i.e. $$QY'=APX'\iff Y'=(Q^{-1}AP)X',$$ and therefore $D=Q^{-1}AP$, or, with your notations, $\; D=C_2AC_1^{-1}$.