I need to prove that $\vec{x}$ is a solution of $A\vec{x}=\vec{b}$:
$$ \begin{vmatrix} 2&-7&-3\\ -4&1&5\\ 1&3&-1\\ \end{vmatrix} \cdot \begin{vmatrix} 5\\ -1\\ 7\\ \end{vmatrix} = \begin{vmatrix} -4\\ 14\\ -5\\ \end{vmatrix} $$
I've done scalar multiplication and have gotten the right answer: $$ \begin{vmatrix} 2\cdot5+(-7)\cdot(-1)+(-3)\cdot7\\ (-4)\cdot5+1\cdot(-1)+5\cdot7\\ 1\cdot5+3\cdot(-1)+(-1)\cdot7\\ \end{vmatrix} = \begin{vmatrix} -4\\ 14\\ -5\\ \end{vmatrix} $$ My question is how would I go the other way, with only $A$ and $\vec{b}$ get $\vec{x}$?
You can reduce the augmented matrix to the Row Reduced Echelon Form (RREF). The RREF will have a row of zeros which means there is a free variable. So the system has infinitely many solutions. You can pick an "appropriate" value of the free variable and show that (5,-1,7) is one of the solutions. This process of solving the system is called Gauss-Jordan elimination.
Since it is a homework question, I cannot give out well-written complete solution. However, you can find many examples on Gauss-Jordan elimination if you google it.