Let $a, b, c$ be positive integers with $0 < a, b, c < 11$. If $a, b, $ and $c$ satisfy \begin{align*} 3a+b+c&\equiv abc\pmod{11} \\ a+3b+c&\equiv 2abc\pmod{11} \\ a+b+3c&\equiv 4abc\pmod{11} \\ \end{align*}then find the sum of all possible values of $abc$.
What I tried:
I only got this far ...
If the equations from top to bottom were labeled $(1),(2),(3)$,
$(1)+3(2)+(3):$
$7a+11b+7c\equiv 11abc\pmod{11}$
$7(a+c)\equiv 0\pmod{11}$.
Since 7 and 11 are coprime, this means $a+c\equiv0\pmod{11}$ or rather just $a+c=11$ in the given range. Note that because 11 is prime, than modular division by any number not divisible by 11 is allowed henceforth.
Back in $(2)$: $3b+11\equiv3b\equiv2abc\pmod{11}$
$3\equiv 2ac\pmod{11}$
$2ac=2a(11-a)=22a-2a^2\equiv-2a^2\equiv3\pmod{11}$
$2a^2\equiv-3\equiv8\pmod{11}$
$a^2\equiv 4\pmod{11}$
I just got this far ...
$3a + b + c \equiv abc \pmod{11}$
$a + 3b + c \equiv 2abc \pmod{11}$
$a + b + 3c \equiv 4abc \pmod {11}$
So
$(3a+b+c)+ 3(a+3b+c) + (a+b+3c)\equiv 11abc$
$7(a+c)\equiv 0\pmod{11}$ As $11$ is prime you can divide by $7$. (This might not be the case if $\gcd(7,11)\ne 1$ but in this case you can.)
$a\equiv -c\pmod {11}$
So we have
$2a+b\equiv -a^2b$
$3b\equiv -2a^2b$
$b-2a \equiv -4a^2b$.
Now $11$ is prime and $b\ne 0$ so $b$ is invertible. (It's not clear in your post if you realize that you CAN"T allways divide by $b$. You can in this case because $b\not \equiv 0$ and $\gcd(b,11) = 1$.)
$3\equiv -2a^2$
$3*5\equiv (-2*5)a^2$
$4 \equiv a^2$ and $a\equiv \pm 2$.
These are the only solutions as $a^2 -4\equiv 0$ and so $(a-2)(a+2)\equiv 0$. Again because $p$ is prime the only $0$ divisors is $0\equiv 11$ so one of $a+2, a-2\equiv 0\pmod{11}$. (This would not be true if $11$ were not prime.)
So $a \equiv \pm 2$ and $c \equiv \mp 2$ so
If $a \equiv 2$ then
$4 + b \equiv -4b\equiv 7b$
$3b\equiv -8b$ (already exhausted)
$b-4\equiv b+7 \equiv -8b\equiv 3b$
ie. $6b \equiv 4$ and $2b\equiv 7$
$2*6b\equiv 2*4$ and $6*2b \equiv 6*7$ so $b\equiv 8$.
SO $a= 2,b= 8, c= 9$ so $abc=144$
or if $a \equiv -2$ then
$-4+b\equiv -4b$ or $5b \equiv 4$ or $9*5b \equiv 9*4$ or $b\equiv 3$
$b+4 \equiv -16b$. or $5b \equiv 4$ or $b\equiv 3$.
So $a=9; b=3;c=2$. And $abc = 54$
So the sum is $144+54 = 198$.