I was solving my linear algebra homework and the activity says:
"Find a linear transformation $T: \mathbb{R}^2 \rightarrow \mathbb{R}^2 $ that transforms the triangle with vertices $A(0;0)$, $B(3;0)$ and $C(5;6)$ into a triangle with vertices $P(1;3)$, $Q(0;0)$ and $S(1;0)$."
I don't know how to find the transformation, what should I do? Solving an equation system, or replacing points or what? Because I just know how to do the opposite thing
We know that $T$ will map $A$ to $Q$ since $T$ must leave the origin fixed. So we have two choices: $$TB=P;\ TC=S$$ or $$TB=S;\ TC=P$$
Now let the matrix of $T$ in terms of the standard basis be $$\pmatrix{a&b\\c&d}$$ Pick one of the two possibilities for $T$ and write down the equations. This will give you four equations in the four unknowns $a,b,c,d.$
It's best to just start out with the equations for $TB$. The $y$ value of $0$ makes it trivial to solve for $a$ and $c$, and then you can plug these into the second set of equations.
EDIT
Another way of looking at it, prompted by the OP's comment. If we write $$\pmatrix{a&b\\c&d}\pmatrix{3\\0}=\pmatrix{3\\1}\\\pmatrix{a&b\\c&d}\pmatrix{5\\6}=\pmatrix{1\\0} $$ that comes to the same thing as writing $$\pmatrix{a&b\\c&d}\pmatrix{3&5\\0&6}=\pmatrix{3&1\\1&0}$$ Since $$\pmatrix{3&5\\0&6}$$ is invertible, we can solve for $a,b,c,d$ no matter what values the matrix on the right-hand side is. So it doesn't matter which point you decide to map $B$ to, you'll get a valid answer.
In this case, there are two correct answers to the question. You should consider giving them both.