I'm having a headache with this...
Given the augmented linear system matrix: $$A = \begin {cases} 1 & 0 & 0 & 2 \\ 0 & a-2 & 0 & 0 \\ 0 & 0 & b + 1 & c \\ 0 & 0 & c & 0 \end {cases}$$
a) Determine for what values of $a,b,c$ the rank is $1$.
b) Determine for what values of $a,b,c$ the rank is $2$.
c) Determine for what values of $a,b,c$ the rank is $3$.
d) Determine for what values of $a,b,c$ the system has no solution.
e) Determine the solution when this system has an unique solution.
First of all, for this sort of problems, I remember seeing someone do it the following way. Here is my attempt at doing this. It took me an unhealthy amount of time, and it's probably wrong because it's HUGE.
... a case tree...
$$\begin {cases} a = 2 \implies \begin {cases} b = -1 \implies \begin {cases} c = 0 \implies \text {rank 1, infinite solutions} \\ c \neq 0 \implies \text {rank 3, NO solution because of 3rd row} \end {cases} \\ b \neq -1 \implies \begin {cases} c = 0 \implies \text {rank 2, infinite solutions} \\ c \neq 0 \implies \text {rank 2, infinite solutions} \end {cases} \end {cases} \\ a \neq 2 \implies \begin {cases} b = -1 \implies \begin {cases} c = 0 \implies \text {rank 2, infinite solutions} \\ c \neq 0 \implies \text {rank 4, NO solutions because of 3rd row} \end {cases} \\ b \neq -1 \implies \begin {cases} c = 0 \implies \text {rank 3, unique solution} \\ c \neq 0 \implies \text {rank 3, unique solution} \end {cases} \end {cases} \end {cases}$$
a) Determine for what values of $a,b,c$ the rank is $1$.
According to my tree... $(2,-1,0)$
b) Determine for what values of $a,b,c$ the rank is $2$.
I guess: $\{ (2,-1,t) \ | \ t \in \mathbb{R} \} \cup \{ (t, -1, 0) \ | \ t \in \mathbb{R} - \{ 2 \} \}$
c) Determine for what values of $a,b,c$ the rank is $3$.
Well, $\{ (2, -1, t) \ | \ t \in \mathbb{R} \} \cup \{ (t,r,s) \ | \ t,r,s \in \mathbb{R} \land t \neq 2 \land r \neq -1\}$
d) Determine for what values of $a,b,c$ the system has no solution.
$\{ (2,-1,t) \ | \ t \in \mathbb{R} - \{ 0 \} \} \cup \{ (t,-1,r) \ | \ t,r \in \mathbb{R} \land t \neq 2 \land r \neq 0 \}$
e) Determine the solution when this system has an unique solution.
I don't get it. For having an unique solution, the rank needs to be $3$. According to my tree, the rank is $3$ when $a \neq 2$ and $b \neq -1$ - but then that means this unique solution depends on $a$ and $b$ (variable), which doesn't make any sense (they should be constant values, right?). I guess that this proves that my case tree is wrong?
It seems pretty likely that this is all wrong and I just wasted my time. Well, I tried. Can you enlighten me here? Surely there is a simple and friendly way to figure this out without using a damned case tree like this other guy did?
Your matrix $$ A = \begin{pmatrix} 1 & 0 & 0 & 2 \\ 0 & a-2 & 0 & 0\\ 0 & 0 & b+1 & c\\ 0 & 0 & c & 0 \end{pmatrix} $$ is almost in row echelon form, so the easiest thing to do is put your augmented matrix in echelon form—remember, row operations preserve rank and the solution space—and see what you can read off. Note, though, that $a$, $b$, $c$ aren't the variables of your system but parameters defining the constant coefficients of the system, i.e., your system of equations is $$ \begin{pmatrix} 1 & 0 & 0 \\ 0 & a-2 & 0\\ 0 & 0 & b+1\\ 0 & 0 & c \end{pmatrix} \begin{pmatrix} x \\ y \\ z \end{pmatrix} = \begin{pmatrix} 2 \\ 0\\ c\\ 0 \end{pmatrix}, $$ with $(x,y,z)^T$ your unknown vector.
First, suppose that $c \neq 0$. Then, if you exchange rows 3 and 4, and then take $$\text{(row 4)} \mapsto \text{(row 4)} - (b+1)c^{-1}\text{(row 3)},$$ you obtain the row echelon form $$ A^\prime = \begin{pmatrix} 1 & 0 & 0 & 2 \\ 0 & a-2 & 0 & 0\\ 0 & 0 & c & 0\\ 0 & 0 & 0 & c \end{pmatrix}. $$ On the one hand, because of the last row, your system is necessarily inconsistent. On the other hand, if $a \neq 2$ then $\operatorname{rank}(A) = 4$, whilst if $a = 2$ then $\operatorname{rank}(A) = 3$.
Now, suppose that $c = 0$. Then your matrix $A$ is already in row echelon form, i.e., $$ A = \begin{pmatrix} 1 & 0 & 0 & 2 \\ 0 & a-2 & 0 & 0\\ 0 & 0 & b+1 & 0\\ 0 & 0 & 0 & 0 \end{pmatrix} $$ On the one hand, observe that your system now is always consistent. On the other hand, if $a \neq 2$ and $b \neq -1$, then $\operatorname{rank}(A) = 3$, and you clearly have a unique solution, if $a \neq 2$ and $b = -1$, then $\operatorname{rank}(A) = 2$, if $a = 2$ and $b \neq -1$, then $\operatorname{rank}(A) =2$, and if $a = 2$ and $b = -1$ then $\operatorname{rank}(A) = 1$.
Collecting all your observations, you therefore find that:
exactly as your decision tree suggested.