
So i did a substitution to solve the system normally, and got
$x=17.67$
$y=9.67$
$z=10.67$
Where I am stuck is how to represent something like this in a vector form, maybe my solution was wrong in the first place.

So i did a substitution to solve the system normally, and got
$x=17.67$
$y=9.67$
$z=10.67$
Where I am stuck is how to represent something like this in a vector form, maybe my solution was wrong in the first place.
For example \begin{align} (1,2,-3)^T \cdot x &= 5 \\ (2,1,-3)^T \cdot x &= 13 \\ (-1,1,0)^T \cdot x &= -8 \end{align} where $x = (x_1,x_2,x_3)^T$.
But more usual is using a matrix equation $$ A x = b $$ with $A$ using the above first argument vectors as row vectors and $b = (5, 13, -8)^T$.
The solution would be $x = A^{-1} b$, if $A$ is invertible. In your case it is not. Your solution works, so there must be infinite many solutions.
$$ (A | b) = \\ \left( \begin{array}{rrr|r} 1 & 2 & -3 & 5 \\ 2 & 1 & -3 & 13 \\ -1 & 1 & 0 & -8 \end{array} \right) \to \\ \left( \begin{array}{rrr|r} 1 & 2 & -3 & 5 \\ 0 & -3 & 3 & 3\\ 0 & 3 & -3 & -3 \end{array} \right) \to \\ \left( \begin{array}{rrr|r} 1 & 0 & -1 & 7 \\ 0 & 1 & -1 & -1 \\ 0 & 0 & 0 & 0 \end{array} \right) $$ This gives the general solution $x = (7 + x_3,-1 + x_3, x_3)^T$ with arbitrary $x_3 \in \mathbb{R}$.