Use the determinant approach to find all values of a and b for the system Ax=k... (system found in picture link)
$$\begin{bmatrix} 1 & 0 & 1 \\ 0 & a & b \\ 0 & b & 1 \end{bmatrix} \begin{bmatrix} x \\ y \\ z \end{bmatrix} = \begin{bmatrix} 2 \\ 1 \\ 2-b \end{bmatrix}$$
First compute $det(A)$ by expanding along the first column: $det(A)=a-b^2$
(a) Provided $det(A)=a-b^2\ne0$ we have that $A$ is invertible, so there will be a unique solution. We can compute the solution by using Cramer's rule, etc.
$$x=\frac{\begin{vmatrix}\color{blue}{2}&0&1\\\color{blue}{1}&a&b\\\color{blue}{2-b}&b&1\end{vmatrix}}{a-b^2}=\frac{2(a-b^2)+(b-a(2-b))}{a-b^2}=\frac{ab+b-2b^2}{a-b^2}$$ and similarly for $y,z$.
(b) There are no solutions if $det(A)=0$ and the equations are inconsistent.
$$det(A)=0 \implies a=b^2$$ so the second and third equations reduce to $$\begin{bmatrix}b^2&b\\b&1\end{bmatrix}\begin{bmatrix}y\\z\end{bmatrix}=\begin{bmatrix}1\\2-b\end{bmatrix}$$
Equations are inconsistent if:
$1\ne b(2-b) \implies b^2-2b+1\ne0 \implies (b-1)^2\ne0 \implies b\ne1,a=b^2$
(c) There is more than one solution if the equations are consistent, i.e. $b=1,a=b^2=1$ for which two solutions are:
$x=1,y=0,z=1$ and $x=2,y=1,z=0$