I am really struggling with this question:
Points $A$, $B$, and $C$ have coordinates $A(1,3)$, $B(5,-1)$, and $C(2,-8)$.
Point $D$ is such that $\vec{AD}=\vec{BC}+2x\vec{AB}+3y\vec{AC}\ =\vec{AB}+2x\vec{AC}+3y\vec{BC}$.
Find the coordinates of $D$.
Here is my attempt, but I have gotten in a muddle, could someone explain how they would tackle this problem?


Your mistake comes from representing point $D$ using variables that already exist in the given equations. Hopefully my solution below is sufficiently clear for you to follow.
Let $D$ be located at $(m,n)$. In the equation that you have, we want to calculate $\vec{AD}$, $\vec{AB}$, $\vec{AC}$, and $\vec{BC}$.
\begin{align} \vec{AD}&=(m,n)-(1,3)=(m-1,n-3)\\ \vec{AB}&=(5,-1)-(1,3)=(4,-4)\\ \vec{AC}&=(2,-8)-(1,3)=(1,-11)\\ \vec{BC}&=(2,-8)-(5,-1)=(-3,-7)\\ \end{align}
We then have \begin{alignat*}{1} \vec{AD}&=\vec{BC}+2x\vec{AB}+3y\vec{AC}\ &=\vec{AB}+2x\vec{AC}+3y\vec{BC}\\ (m-1,n-3)&=(-3,-7)+2x(4,-4)+3y(1,-11)\ &=(4,-4)+2x(1-11)+3y(-3,-7)\\ (m-1,n-3)&=(8x+3y-3,-8x-33y-7)\ &=(2x-9y+4,-22x-21y-4) \end{alignat*}
Using the second and third columns, we have the system of equations
$$\left\{\begin{align} 2x-9y+4&=8x+3y-3\\ -22x-21y-4&=-8x-33y-7 \end{align}\right.$$ Simplifying, this gives us $$\left\{\begin{align} 0&=6x+12y-7&\qquad (1)\\ 0&=14x-12y-3&\qquad (2) \end{align}\right.$$
Then, \begin{align} (1)+(2):\quad 0&=20x-10\\ x&=\frac12 \end{align} and \begin{align} (1):\quad 0&=6(\frac12)+12y-7\\ 0&=12y-4\\ y&=\frac13\end{align}
Finally, using the first and second columns in the three-column set of equations above, we have \begin{align} (m-1,n-3)&=(8(\frac12)+3(\frac13)-4,-8(\frac12)-33(\frac13)-7)\\ (m-1,n-3)&=(4+1-3,-4-11-7)\\ (m-1,n-3)&=(2,-22)\end{align}
Simultaneously solve \begin{align} m-1&=2\\ m&=3\\ n-3&=-22\\ n&=-19\end{align} Therefore, point $D$ is located at $(3,-19)$.