Find constants $a ,\ b ,\ c \ $ such that the equation of the circle, $x^2+y^2+ax+by=c$, contains the points $(6,8)$, $(8,4)$, and $(3,9)$.
Use the points to create a system:
$\begin{array}{rcrcrcrcr} 6^2 & + & 8^2 & + & 6a & + & 8b & = & c \\ 8^2 & + & 4^2 & + & 8a & + & 4b & = & c \\ 3^2 & + & 9^2 & + & 3a & + & 9b & = & c \\ \end{array}$
Place it into an augmented matrix and perform row operations to reduce:
$\displaystyle \left[ \begin{array}{rr|rcr} 8 & 6 & c &-& 100 \\ 4 & 8 & c &-& 80 \\ 3 & 9 & c &-& 90 \\ \end{array} \right] $ ($R_2-R_3$)
$\displaystyle \left[ \begin{array}{rr|rcr} 8 & 6 & c &-& 100 \\ 1 & -1 & 10 \\ 3 & 9 & c &-& 90 \\ \end{array} \right] $ ($R_1-R_3$)
$\displaystyle \left[ \begin{array}{rr|rcr} 5 & -3 & -10 \\ 1 & -1 & 10 \\ 3 & 9 & c &-& 90 \\ \end{array} \right] $ ($R_1-3R_2; \frac{1}{2}R_1; -R_2)$
$\displaystyle \left[ \begin{array}{rr|rcr} 1 & 0 & -20 \\ -1 & 1 & -10 \\ 3 & 9 & c &-& 90 \\ \end{array} \right] $ ($R_2+R_1$)
$\displaystyle \left[ \begin{array}{rr|rcr} 1 & 0 & -20 \\ 0 & 1 & -30 \\ 3 & 9 & c &-& 90 \\ \end{array} \right] $ ($R_1-R_1$)
This gives $c$ such that the given circle has a negative radius, but surely that can't be. Where is my error?
As noted in the comment of thanasissdr, the third row of your starting matrix is wrong. It must be $9,3,c-90$
Anyway, solving correctly the system, you find: $a=-6$,$b=-8$ and $c=0$.
This means that the circle has center $C=(\alpha,\beta)=(-a/2,-b/2)=(3,4)$ and radius $r=\sqrt{\alpha^2+\beta^2+c}=5$.