Consider the basis $\{\vec{p},\vec{q}\}$ where $\vec{p}=(1,1)$ and $\vec{q}=(-1,0)$. Let $T:\mathbb{R}^2\to\mathbb{R}^2$ be the linear operator such that $T(\vec{p})=(1,-2)$ and $T(\vec{q})=(4,1)$. Then find a formula for $T(x_1,x_2)$ and use this formula to find $T(5,-7)$.
I figured this might make the transformation matrix $A$ be $ A = \left[ \begin{array}{rr} 1 & 4 \\ -2 & 1 \\ \end{array} \right]$ such that $T_A(\vec{v}) = A\vec{v}$, but this doesn't seem to pan out.
If we interpret the question as if all vectors are expressed in the standard basis, than we have to find a matrix : $$ A_s=\begin{bmatrix} a&b\\ c&d \end{bmatrix} $$ such that $$ \begin{bmatrix} a&b\\ c&d \end{bmatrix} \begin{bmatrix} 1\\ 1 \end{bmatrix}= \begin{bmatrix} 1\\ -2 \end{bmatrix} \quad \land \quad \begin{bmatrix} a&b\\ c&d \end{bmatrix} \begin{bmatrix} -1\\ 0 \end{bmatrix}= \begin{bmatrix} 4\\ 1 \end{bmatrix} $$ this gives the system: $$ \begin{cases} a+b=1\\c+d=-2\\-a=4\\-c=1 \end{cases} $$ with solutions $a=-4,b=5,c=-1,d=-1$ and the matrix $A$ is: $$ A_s=\begin{bmatrix} -4&5\\ -1&-1 \end{bmatrix} $$
IN this interpretation the fact that $\{\vec p,\vec q\}$ is a basis is true but not relevant.
If the components of $T(\vec p)$ and $T(\vec q)$ in OP are expressed in the basis $t=\{\vec p,\vec q\}$, than ,since in this basis we have $\vec p=[1,0]^T$ and $\vec q=[0,1]^T$, than it is represented , in this basis, by the matrix: $$ A_t=\begin{bmatrix} 1&4\\ -2&1 \end{bmatrix} $$ but note that this transformation is different from the other one.