This is my question:
Does there exist a linear transformation $T:\mathbb R^2\to \mathbb R^3$ such that $$T([1,0]) = [1,2,3] , T([0,1]) = [-1,2,1], \text{ and }T([5,1]) = [4,12,2] ?$$
How do I set up an augmented matrix for this? I tried doing it without setting up an augmented matrix (and then solving it using Gaussian elimination), and got
\begin{pmatrix} 1 &-1 \\ 2 & 2 \\ 3& 1\end{pmatrix}
as my answer. But that doesn't seem right because the last $T([5,1])$ doesn't give me $[4,12,2]$.
Edit: turns out my transformation matrix does gave me [4,12,2], so I know my answer is right, but is there any other way to solve this by gaussian elimination (when there's more complicated inputs unlike [1,0] or [0,1])?
No, your transformation matrix does not give $[4,12,2]$ for $T([5,1])$. You were correct the first time. You’re making some error in the matrix multiplication. At any rate, the product of this matrix with $[5,1]$ is just a repackaging of $$T([5,1]) = 5 T([1,0])+T([0,1]) = 5[1,2,3]+[-1,2,1] = [4,12,16],$$ which is what you probably got the first time. Indeed, it’s clear from the given conditions that there is no such linear transformation since linearity is violated by them. I expect that this was the point of the exercise—to get you to check this before doing any other work. It’s a simple sanity check that you should do whenever you’re faced with a problem of this type.
Now, you could could certainly set up a matrix with unknown entries and use the three given conditions to generate a system of linear equations in these unknowns, but you would quickly find that the system is inconsistent. However, the transformation matrix can be written down directly from the given conditions. Recall that the columns of a transformation matrix are the images of the basis vectors, which you’ve been given. Therefore, the only possiblity for the matrix of $T$ is $$\begin{bmatrix}1&-1\\2&2\\3&1\end{bmatrix},$$ but, as you correctly determined the first time, this gives the wrong result for $[5,1]$, so once again the correct conclusion is that there is no such linear transformation.