Given that the rank of $\mathbf A=\left(\begin{smallmatrix}1&2&1&1\\3&a&4&3\\5&8&6&b\end{smallmatrix}\right)$ is $2$, find $a$ and $b$.
I used matrix transformation and got $$\left(\begin{smallmatrix}1&2&1&1\\3&a&4&3\\5&8&6&b\end{smallmatrix}\right)\longrightarrow\left(\begin{smallmatrix}1&2&1&1\\0&a-6&1&0\\0&-2&1&b-5\end{smallmatrix}\right)\longrightarrow\begin{pmatrix}1&0&2&b-4\\0&0&\frac{a-4}2&\frac{(a-6)(b-5)}2\\0&-2&1&b-5\end{pmatrix}.$$ I don't know how to continue.
First, notice that the first and third columns $C_1 = \begin{pmatrix} 1 \\ 3 \\ 5 \end{pmatrix}$ and $C_3 = \begin{pmatrix} 1 \\ 4 \\ 6 \end{pmatrix}$ of $A$ are not colinear.
Hence, the rank of $A$ is at least $2$ in every case. For the rank to be exactly equal to $2$, you need the second and fourth columns $C_2$ and $C_4$ to belong to the space generated by $C_1$ and $C_3$.
So you need $\det(C_2, C_1, C_3)=0$ and $\det(C_4, C_1, C_3)=0$.
But $$\det(C_2, c_1, C_3)= \begin{vmatrix} 2 & 1 & 1 \\ a & 3 & 4\\ 8& 5 & 6\end{vmatrix} = 4-a$$
and $$\det(C_4, c_1, C_3)= \begin{vmatrix} 1 & 1 & 1 \\ 3 & 3 & 4\\ b& 5 & 6\end{vmatrix} = b-5$$
So finally, $A$ has rank $2$ if and only if $a=4$ and $b=5$.