So for my question, I am given a system equation as below:
$$\begin{array}{cccc|c} 1 &-1 & 2 & 3 & -2\\ -3 & 4 & 1 & -6 & 5 \\ 2 & 1 & a^2+9 & a+11 & b \\ \end{array}$$
And the problem requires me to find all the possible values of $a$ and $b$ such that the system has i) no solutions and ii) has infinietely many solutions with two free variable.
First of all, I tried to reduce the matrix above into the row echelon form using the row operations $R_{2}\to R_{2}+3R_{1}$, $R_{3}\to R_{3}-2R_{1}$, and $R_{3}\to R_{3}-3R_{2}$, which gives me the matrix below:
$$\begin{array}{cccc|c} 1 &-1 & 2 & 3 & -2\\ 0 & 1 & 7 & 3 & -1 \\ 0 & 0 & a^2-16 & a-4 & b+7 \\ \end{array}$$
Using this matrix, I get the answer for i) $a=4$ and $b \neq -7$ , ii) $a=4$ and $b\neq7$.
This is because the quadratic $a^2-16$ can give two solutions, either $4$ and $-4$ which leads us to reject the solution of $x=-4$
However, I tried to experiment and further reduce this row echelon form using the one more row operation $R_{3}\to R_{3}- \frac{a^2-16}{7} R_{2}$, to get:
$$\begin{array}{cccc|c} 1 &-1 & 2 & 3 & -2\\ 0 & 1 & 7 & 3 & -1 \\ 0 & 0 & 0 & a-4-(\frac{a^2-16}{7}*3) & b+7+\frac{a^2-16}{7} \\ \end{array}$$
I believe that the form above is also a row echelon form, as each pivot is one column to the right of the pivot of the row above. But when solving this quadratic, I get one extra solution, which is $a=-5/3$ from solving the equation $a-4-(\frac{a^2-16}{7}*3)=0$.
This means that the answer for question i) becomes $a=4$ and $b\neq-7$ and $a=-\frac{5}{3} $and $ b\neq\frac{46}{9}$.
How is it possible can I get an extra solution just from using the different row echelon form?