Let's have $\ M= \left[ {\begin{array}{ccccc} 1 & \alpha & -1 & 2 \\ 2 & -1 & \alpha & 5 \\ 1 & 10 & -6 & 1 \\ \end{array} } \right]$.
Using Gaussian elimination I've come to have the next matrix $\ M'= \left[ {\begin{array}{ccccc} 1 & \alpha & -1 & 2 \\ 0 & 1 & \frac{-5}{10- \alpha} & \frac{-1}{10 - \alpha} \\ 0 & 0 & \frac{-5(1+2\alpha)}{10- \alpha}+\alpha +2 & 1 \\ \end{array} } \right]$
So:
- $\alpha \neq 10$
- $\frac{-5(1+2\alpha)}{10- \alpha}+\alpha +2 = 1$ and then i calculate the values of $\alpha$
Is it ok what I did? Is the rank always going to be 3 no matter the value of $\alpha \neq 10$ ? What if $\alpha = 10$ ?
Since it is not a square matrix this is the only way to find out the rank, or is there another way?
A good strategy for these problems is to do as many operations as you can without dividing by an expression involving a variable.
In particular, I find: $$ \pmatrix{ 1 & \alpha & -1 & 2 \\ 2 & -1 & \alpha & 5 \\ 1 & 10 & -6 & 1 \\ } \to \pmatrix{ 1 & \alpha & -1 & 2 \\ 0 & -1 - 2\alpha & 2+\alpha & 1 \\ 0 & 10-\alpha & -5 & -1 \\ } $$ The rank of this matrix is always at least $2$, since the first and last rows are linearly independent. We can only have rank exactly two if the matrix $$ \pmatrix{-1-2\alpha & 2+\alpha & 1\\ 10-\alpha & -5 & -1} $$ has rank $1$. That is, we can only achieve rank $2$ if the first and second columns are multiples of the third. In other words, we must have $$ -(-1-2\alpha) = 10 - \alpha \implies \alpha = 3\\ (-(-5)) = 2 + \alpha \implies \alpha = 3 $$ So indeed, when $\alpha = 3$, the matrix has rank $2$. In all other cases, the matrix has rank $3$.
A more computational approach is to note that permuting the columns (or more generally "applying column operations") doesn't change the rank. So, we can have $$ \pmatrix{ 1 & \alpha & -1 & 2 \\ 2 & -1 & \alpha & 5 \\ 1 & 10 & -6 & 1 \\ } \to \pmatrix{ 1 & \alpha & -1 & 2 \\ 0 & -1 - 2\alpha & 2+\alpha & 1 \\ 0 & 10-\alpha & -5 & -1 \\ } \\ \to \pmatrix{ 1 &2 & \alpha & -1 \\ 0 &1 & -1 - 2\alpha & 2+\alpha \\ 0 & -1 & 10-\alpha & -5 \\ } \\ \to \pmatrix{ 1 &2 & \alpha & -1 \\ 0 &1 & -1 - 2\alpha & 2+\alpha \\ 0 & 0 & 9-3\alpha & -3 + \alpha \\ } $$ It is clear that this matrix will have full rank whenever $\alpha \neq 3$, and rank $2$ when $\alpha = 3$.