$$\left( {\begin{array}{*{20}{c}} a & b & {a + b} \\ {2a} & {a + b} & {a - b} \\ a & a & {2a - b} \\ \end{array}} \right) \simeq \left( {\begin{array}{*{20}{c}} a & b & {a + b} \\ 0 & {a - b} & { - a - 3b} \\ 0 & 0 & {2a + b} \\ \end{array}} \right)$$
As you can see, I row-reduced the matrix to the final form on the right side of the equation. The exercise answer is: $a\ne b, a\ne 0,b\ne -2a$
I don't understand why $a\ne b, a\ne 0$ are needed. I doubled checked my reduction process.
On
All the pivots have to be different than zero.
That means all the number on the diagonal, hence the answer
On
If $a = 0$ or $b = a$, it's obvious the matrix has at least one zero in its diagonal. Considering the fact that it's a top-triangular matrix, it will obviously make its determinant null, thus showing that $\operatorname{rank}(A)$ is smaller than $3$.
On
\begin{pmatrix} \begin{array}{*{20}{c}} a & b & {a + b} \\ 0 & {a - b} & { - a - 3b} \\ 0 & 0 & {2a + b} \\ \end{array} \end{pmatrix} This matrix has rank $3$ iff all three columns are linearly independent.
If $a=b \neq 0$, then the first and second column are equal hence linearly dependent.
If $a=0$, then the first column is $\bar{0}$ in which case the rank of the matrix is $2$.
The rank of a $3\times3$ matrix is $3$ if and only if its determinant is nonzero. You have brought your matrix into triangular form by row operations (which do not change the rank), so its rank is (still) equal to $3$ if and only if the determinant of the triangular matrix is nonzero. This determinant is easily computed to be $a(a-b)(2a-b)$. Can you take it from there?
If you don't want to use determinants, you can easily argue for a triangular matrix that the first column to be in the span of its preceding columns, if such a column exists, is the first column to have a zero diagonal entry. So the rank is $3$ if and only if all diagonal entries are nonzero. (The argument I alluded to is: if the diagonal entry is nonzero, then the column cannot be in the span of the previous ones, as they all have zeros in that row; but if the diagonal entry is zero and the previous columns are independent, then the new column must be in their span for dimension reasons.)