I am to calculate the value of this matrix
$$ \begin{bmatrix} 1 & 1 & 1 \\ 1 & 1 & a \\ 1 & b & 1 \end{bmatrix} $$
I do a basic transformation to
$$ \begin{bmatrix} 1 & 1 & 1 \\ 0 & -b+1 & 0 \\ 0 & 0 & -a+1 \end{bmatrix} $$
We have the "stairs" in the left corner. How could we proceed to get the right answer? I know how to calculate basic matrix , but variables $a,b$ in it makes it hard for me to understand it
What you have to do, now that you have a triangularized matrix, is to "discuss" according to the values of $a$ and $b$:
if $a\neq1$ and $b\neq1$, the determinant is $\neq 0$: thus rank=3.
if $a=1$ and $b\neq1$, the determinant is zero, thus rank $<3$ but the two first vectors are independant, thus rank=2.
if $a\neq1$ and $b=1$, same conclusion, this time with column 1 and 3.
if $a=1$ and $b=1$, the rank drops to 1 because all the columns are multiple of the first one.