The matrix is as follows: $$ \left[ \begin{array}{ccc|cc} a&0&b&2\\ a&a&4&4\\ 0&a&2&b \end{array} \right] $$
I need to find when this matrix will have one, infinitely many, and no solutions by expressing condition on a, b and c
I did put the matrix in reduced row echelon form as follow:
$$ \left[ \begin{array}{ccc|cc} 1&0&0&\frac{2-b}{a}\\ 0&1&0&\frac{b-2}{a}\\ 0&0&1&\frac{2-b}{a} \end{array} \right] $$
but now I'm stuck about how to express the conditions on a, b, and c
Reduction should have gotten you here:
$$ \left[ \begin{array}{ccc|cc} a&0&b&2\\ 0&a&4-b&2\\ 0&0&b-2&b-2 \end{array} \right] $$
This is far enough to analyze the different cases:
We must have a leading term for each column. Therefore, $b \neq 2$ and $a \neq 0$.
For this to occur we must have a row where the left side is all zeros, but the right side is not. This can occur if:
This occurs when we have a free variable. This is achieved by getting a whole row (including the right side) or column to be zeros: