This is a problem from my signals and systems class. The question is asking for what values of b1, b2, b3 such that the system is controllable (determinant is 0). Image of question
Is it even possible to take the determinant of a non-square matrix? In this problem I am quite confused as how to even begin to evaluate the determinant of this rectangular matrix.
In addition how come the answer boils down to the determinant of a 2 x 2 matrix?
I have attached a picture of the solution here.
Controllability is not about the "determinant" being equal to zero. Indeed, as you've correctly stated, there is no well-established concept of the determinant of a non-square matrix (although certain generalizations exist). What we're interested in is the rank of the controllability matrix, and in particular we call the system controllable if and only if its controllability matrix is surjective, i.e. has full row rank.
The question now becomes: What do determinants have to do with determining row rank?
Well, we know that the row rank and the column rank of a matrix are always equal. You have a $3\times 6$ matrix, so full row rank means you require rank $3$. This means you need to find $3$ linearly independent columns in order to establish full rank.
In your solution you've already established that columns $1,3,5$ are linearly independent from $2,4,6$ (which are in turn all linearly dependent). This means you've already found two linearly independent columns, say $1$ and $2$. If you can further establish that two of the columns amongst $1,3,5$ are linearly independent, then you would've found the required three linearly independent columns.
So how do we do that? Well a simple way is to use determinants. In your controllability matrix, columns $1$ and $3$ are $$\begin{pmatrix}b_1 \\ 0 \\b_3\end{pmatrix},\ \ \ \text{and}\ \ \ \begin{pmatrix}b_1 - b_3 \\ 0 \\ 3b_3 - 2b_1\end{pmatrix}.$$ Since we are interested in the linear independence of these two vectors, we can ignore the middle entry and put the vectors into a $2\times 2$ matrix $$\begin{pmatrix}b_1 & b_1-b_3 \\ b_3 & 3b_3 - b_1\end{pmatrix}.$$ Then the two vectors will be linearly independent if and only if this auxillary $2\times 2$ matrix were invertible, i.e. if the determinant were non-zero. This is exactly what your solution is doing. It is choosing to make columns $1$ and $3$ linearly independent, so that we have a full set of three linearly independent columns, i.e. $1,2,3$. This would make the system controllable. Note that your solution is not looking for values of $b_1,b_2,b_3$ which make the determinant zero; it's looking for values which make the determinant non-zero so that the resulting matrix would have the requisite rank.
In general, to establish that a matrix has rank (at least) $r$, you would need to find a non-vanishing order $r$ minor, which means taking determinants of a square submatrix. This is effectively what your solution is doing, but with various shortcuts which obscure the process.