I've been given three matrices $A, B \ \& \ C$ which are defined as follows:
$$ A = { \left[ \begin{array}{ccc} b & 5 & 8 \\ c & 1 & 3 \\ a & 4 & 3 \\ \end{array} \right] },\ B = { \left[ \begin{array}{ccc} 7 & 4 & 2 \\ 5 & 5 & -1 \\ -2 & -a & -b \\ \end{array} \right] }, \ C = { \left[ \begin{array}{ccc} 3 & c & 5 \\ 1 & b & 4 \\ 2 & a & 1 \\ \end{array} \right] } $$
First, I've been asked to find the determinant of each matrix in terms of the parameters, and I've done it like this:
$ |A| = b \begin{vmatrix} 1 & 3 \\ 4 & 3 \\ \end{vmatrix} - 5 \begin{vmatrix} c & 3 \\ a & 3 \\ \end{vmatrix} + 8 \begin{vmatrix} c & 1 \\ a & 4 \\ \end{vmatrix} = \ b(3-12)-5(3c-3a)+8(4c-a) = \\ -9b - 15c + 15a + 32c - 8a = \mathbf{7a - 9b + 17c} $
$ |B| = 7 \begin{vmatrix} 5 & -1 \\ -a & -b \\ \end{vmatrix} - 4 \begin{vmatrix} 5 & -1 \\ -2 & -b \\ \end{vmatrix} + 2 \begin{vmatrix} 5 & 5 \\ -2 & -a \\ \end{vmatrix} = 7(-5b-a)-4(-5b-2)+2(-5a+(-10)) = -35b - 7a + 20b + 8 - 10a + 20 = \mathbf{-17a -15b + 28} $
$ |C| = 3 \begin{vmatrix} b & 4 \\ a & 1 \\ \end{vmatrix} - c \begin{vmatrix} 1 & 4 \\ 2 & 1 \\ \end{vmatrix} + 5 \begin{vmatrix} 1 & b \\ 2 & a \\ \end{vmatrix} = \ 3(b - 4a) - c(1 - 8) + 5(a - 2b) = \\ 3b - 12a – c + 8c + 5a - 10b = \mathbf{-7a -7b + 7c} $
So after this, I have to calculate the value of the parameters to make all three of the matrices not invertible, so I assume $|M| = 0$ has to be true for each matrix. But I don't know how to proceed.
I thought of defining a system of linear equations using the three of them equaled to $ 0 $ but as $|B|$ has no $c$ term I don't know how to acomplish this.
Your determinants are right. Next you set them equal to $0$. You do not ignore the number 28. You put it on the RHS as mentioned in the comments.
$$\left( \begin{array}{ccc|c} 7 & -9 & 17 & 0\\ -17 & -15 & 0 & -28\\ -7 & -7 & 7 &0 \\ \end{array} \right)$$
Now you can use the Gaussian elimination algorithm to obtain the solution. Here are the two steps.
Step 1
Multiply row 1 by $17/7$ and add to row 2.
Add row 1 to row 3.
$$\left( \begin{array}{ccc|c} 7 & -9 & 17 & 0\\ 0 & -258/7 & 289/7 & -28\\ 0 & -16 & 24 &0 \\ \end{array} \right)$$
Step 2
Multiply row 2 by $-56/129$ and add to row 3.
$$\left( \begin{array}{ccc|c} 7 & -9 & 17 & 0\\ 0 & -258/7 & 289/7 & -28\\ 0 & 0 & 784/129 &1568/129 \\ \end{array} \right)$$