I have matrix A = \begin{bmatrix}1&2&3&4&5\\2&4&3&5&4\\3&6&5&8&7\end{bmatrix}
and\vec b = \begin{bmatrix}1\\2\\3\end{bmatrix}
I expanded the matrix by adding the elements of the vector to the matrix. At first I changed the form of the matrix to get a reduced form.
1.R2 - 2xR1
2.R3 - 3xR1
3.R1 + R2
4.R2-R3
5.R3 + 4x R2
Performing these operations, I have the following matrix:
\begin{bmatrix}1&2&0&1&-1&1\\0&0&1&1&2&0\\0&0&0&0&0&0\end{bmatrix} Now I need to remove the last row because it contains only zeros. After that I need to expand the matrix to a n x n+1 matrix. \begin{bmatrix}1&2&0&1&-1&1\\0&0&1&1&2&0\\0&0&0&0&0&0\\0&0&0&0&0&0\end{bmatrix} At this point, I'm rather confused, because it says, that some zeros need to be replaced by a -1 but I don't know which one.
Let \begin{align*} A &= \left[\begin{array}{rrrrr} 1 & 2 & 3 & 4 & 5 \\ 2 & 4 & 3 & 5 & 4 \\ 3 & 6 & 5 & 8 & 7 \end{array}\right] & \vec b &= \begin{bmatrix}1\\2\\3\end{bmatrix} \end{align*}
You have correctly shown that $\DeclareMathOperator{rref}{rref}$ $$ \rref \left[\begin{array}{rrrrr|r} 1 & 2 & 3 & 4 & 5 & 1 \\ 2 & 4 & 3 & 5 & 4 & 2 \\ 3 & 6 & 5 & 8 & 7 & 3 \end{array}\right] = \left[\begin{array}{rrrrr|r} 1 & 2 & 0 & 1 & -1 & 1 \\ 0 & 0 & 1 & 1 & 2 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \end{array}\right] $$ This tells us that $A\vec x=\vec b$ if and only if $$ \begin{array}{rcrcrcrcrcrcrcrc} x_1 &+& 2\,x_2&+&0\,x_3&+&x_4&+&(-1)\,x_5&=& 1 \\ 0\,x_1 &+& 0\,\,x_2&+&x_3&+&x_4&+&2\,x_5&=& 0 \end{array} $$ That is, $A\vec x=\vec b$ if and only if $$ \vec x= \begin{bmatrix} x_1\\x_2\\x_3\\x_4 \end{bmatrix} = \begin{bmatrix} 1-2\,x_2-x_4+x_5\\ x_2\\ -x_4-2\,x_5\\ x_4\\ x_5 \end{bmatrix}= \begin{bmatrix} 1\\0\\0\\0 \end{bmatrix} +x_2\begin{bmatrix} -2\\1\\0\\0\\0 \end{bmatrix}+ x_4 \begin{bmatrix} -1\\0\\-1\\1\\0 \end{bmatrix}+ x_5 \begin{bmatrix} 1\\0\\-2\\0\\1 \end{bmatrix} $$