Given
$$\begin{pmatrix}0&1&2\\ 4&3&a\\ 1&2&4\end{pmatrix}\begin{pmatrix}x\\ y\\ z\end{pmatrix}=\begin{pmatrix}1\\ b\\ 4\end{pmatrix}$$
I need to find for what values $a,b \in \mathbb{F}_{11}$ there are solutions and what the general solution is (without determinan).
I created the following matrix
$$\begin{pmatrix}0&1&2&|1\\ 4&3&a&|b\\ 1&2&4&|4\end{pmatrix}$$
and after Gaussian elimination I got
$$\begin{pmatrix}4&3&a&|b\\ 0&5&10&|5\\ 0&0&a+5&|b\end{pmatrix}$$
How to find witch values of $a,b$ this system is trace?
Writing $Ax=b$ for your system of linear equations, the determinant of $A$ is given by $$ \det(A)=2(2a-27). $$ Over the field $\mathbb{F}_{11}$ the determinant is non-zero if and only if $a\neq 8$ (we could write $a\neq -3=8$ as well). Then the unique solution is given by $$ x=A^{-1}b=\frac{1}{a+3}\begin{pmatrix} 9(a + 7b + 7) \cr 3(a + 9b + 5) \cr b + 10 \end{pmatrix}. $$ In case that $a=8$, we get a $1$-parameter solution, because the rank of $A$ is then equal to $2$. More precisely, we obtain $b=1$ and $x=(x_1,x_2,x_3)$ is a the general solution with $x_1=8(x_3 + 8)$ and $x_2=5(x_3 + 5)$. The parameter $x_3$ is free, and can be any element of the finite field.