Following is augmented matrix which has been reduced to row echelon form by using row operations. So when I convert it to system of equations I would get 3 equations with 5 unknowns. Is it possible to find values of 5 unknowns in 3 equations? Is it true that at most one can solve equations with 3 unknowns in 3 equations?
$$ \left[ \begin{array}{rrrrr|r} 1 & 7& -2 & 0 & -8 & -3 \\ 0 & 0 & 1 & 1 & 6 & 5 \\ 0 & 0 & 0 & 1 & 3 & 9 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ \end{array} \right] $$
On
Is it possible to find values of 5 unknowns in 3 equations?
Yes, the solutions of your system form a 2 dimensional subspace of $F^5$, where $F$ is your field, e.g. $F = \mathbb{R}$.
$$ \left[ \begin{array}{rrrrr|r} 1 & 7 & -2 & 0 & -8 & -3 \\ 0 & 0 & 1 & 1 & 6 & 5 \\ 0 & 0 & 0 & 1 & 3 & 9 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ \end{array} \right] \to \left[ \begin{array}{rrrrr|r} 1 & 7 & 0 & 2 & 4 & 7 \\ 0 & 0 & 1 & 1 & 6 & 5 \\ 0 & 0 & 0 & 1 & 3 & 9 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ \end{array} \right] \to \left[ \begin{array}{rrrrr|r} 1 & 7 & 0 & 0 & -2 &-11 \\ 0 & 0 & 1 & 0 & 3 & -4 \\ 0 & 0 & 0 & 1 & 3 & 9 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ \end{array} \right] $$ E.g. $$ x_5 = t \\ x_4 + 3t = 9 \iff x_4 = 9 - 3t \\ x_3 + 3t = -4 \iff x_3 = -4 - 3t \\ x_2 = s \\ x_1 + 7s - 2t = -11 \iff x_1 = -11 - 7s + 2t $$ then $$ x = \{ (-11-7s+2t, s, -4 - 3t, 9-3t, t) \mid s, t \in F \} $$
To get a unique answer to a system of linear equations you require as many linearly independent equations as you have variables. So in your case you will not be able to get a unique solution. You will be able to express the answer in terms of two parameters instead.