The task is to find the variables a, b so that, the planes $E1, E2 $ and $E3$,
1) have no common point
2) have a common point
The equations are:
$$E1: 1x - 1y - 1z = -2$$
$$E2: 3x +1y -1z = b$$
$$E3: ax + 8y+2z =7$$
I think task 1 has to solved like this: Means one line gives no sense:
$$ \begin{pmatrix} x & y & z & ? \\ x & y & z & ? \\ 0 & 0 & 0 & 4\\ \end{pmatrix}$$
2 like this so that there is a unique solution:
$$ \begin{pmatrix} 1 & 0 & 0 & ? \\ 0 & 1 & 0 & ? \\ 0 & 0 & 1 & ?\\ \end{pmatrix}$$
Are my two approaches correct? And how should i start? Thanks
Writing in matrix form and row--reducing, $$\eqalign{\pmatrix{1&-1&-1&|&-2\cr 3&1&-1&|&b\cr a&8&2&|&7\cr} &\sim\pmatrix{1&-1&-1&|&-2\cr 0&4&2&|&b+6\cr 0&8+a&2+a&|&7+2a}\cr &\sim\pmatrix{1&-1&-1&|&-2\cr 0&4&2&|&b+6\cr 0&0&2a-8&|&-20+2a-8b-ab\cr}\ .\cr}$$ The planes have no common point if these equations have no solution, which occurs when $$2a-8=0\ ,\quad -20+2a-8b-ab\ne0\ .$$ They have a unique common point if $$2a-8\ne0\ ,$$ and more than one common point (in fact, infinitely many) if $$2a-8=0\ ,\quad -20+2a-8b-ab=0\ .$$ You can now solve these equations to find the actual values of $a$ and $b$ in the various cases.
Comment. Unless your instructor insists on it, there is no need to reduce the left hand side to just $1$s and $0$s.