Problem. Let $V = {\vec{v_1} = (5, 3, 1), \vec{v_2}= (1, −3, −2), \vec{v_3} = (1, 2, 1)}$ base of $\mathbb{R^3}$ and let $f$ be the linear operator on $\mathbb{R^3}$ defined by $f(\vec{v_1}) = (−2, 1, 0)$, $f(\vec{v_2}) = (−1, 3, 0)$, $f(\vec{v_3}) = (−2, −3, 0)$. Calculate the matrix of $f$ in the base $V$;
How I was taught the lesson. Now the issue is that in my book and all of my notes they always define the matrices of change of base in this way: $$y'=F_v x'$$ for the matrix $F_v$ of $f$ in a base $V$ and the matrix $P$ to change basis from the canonical to the base $V$: $$x'=Px$$ (I'm using ' for the base $V$ and no ' for the canonical).
How I approached the problem. So I continued doing the problem: $$x'=Px\Rightarrow x=P^{-1} x'$$ since we were given the $y'$ in terms of $x$ (the canonical base) because $f(\vec{v_1})=-2\vec{e_1}+\vec{e_2}$, $f(\vec{v_2})=-\vec{e_1}+2\vec{e_2}$ and $f(\vec{v_3})=-2\vec{e_1}-3\vec{e_2}$, we have found that matrix $M$ such that $y'=Mx$, so then: $$y'=Mx=M(P^{-1}x')=MP^{-1}x'=F_v x'$$ which I found the solution to be $F_v=MP^{-1}$ but my book now says it is $P^{-1}M$, how is that possible?
The way they approached the problem. The same was as I did, but instead of $y'=Mx$ they did $y'=xM$, and likewise with $x'=xP$ changing the order of the matrix and putting it in the right? And they arrive to $F_v=P^{-1}M$.
My doubt. What happened? Have I done something wrong or is it just that both answers are valid? And why did they put the matrix on the right and not on the left, does it not matter as long as you are consistent with it?
$\require{AMScd}$ \begin{CD} V @>F>> V\\ @V P V V \huge\searrow\small M @VV P V\\ \Bbb R^3 @>>\phi> \Bbb R^3 \end{CD}
$x:$ Old coordinates in domain, $y:$ Old coordinates in co-domain, $x':$ New coordinates in domain, $y':$ New coordinates in co-domain.
We have $$y'=Fx'\tag1$$ where $F$ is the transformation matrix of the given transformation $\phi$ in the new coordinates. We want to find $F$.
Remember that old coordinates and new coordinates are connected via transition matrix $P$ in the following way: $x=Px'$ in the domain and $$y=Py'\tag2$$ in the co-domain, where $\large P=[v_1|v_2|v_3]$ is the transition matrix.
On the other hand we have $$y=Mx'\tag3$$ where $\large M=[f(v_1)|f(v_2)|f(v_3)]$ is the tranformation matrix of the same transformation with new coordinates in the domain but old coordinates in the co-domain.
From $(2)$ and $(3)$, $y'=P^{-1}Mx'$. From $(1)$ and this, we have $Fx'=P^{-1}Mx'.$ Therefore $F=P^{-1}M$.
OP's Example: $P=\left[\begin{matrix}5&1&1\\3&-3&2\\1&-2&1\end{matrix}\right]$ and $M=\left[\begin{matrix}-2&-1&-2\\1&3&-3\\0&0&0\end{matrix}\right]$. With some help of computer, we find $$F=P^{-1}M=\left[\begin{matrix}-5&-10&7\\6&13&-10\\17&36&-27\end{matrix}\right].$$