I was given the question Solve this formula for X1 and X2 over Z7 (mod7)
$$ 3x^2+x+1 = 0 $$
So I went on and converted it to the quadratic formula over Z7 like and solved it like so: $$ \frac{6\pm \sqrt{1+3*3*1}}{6} = \frac{-1\pm \sqrt{1-4*3*1}}{6} = \frac{-1\pm \sqrt{3}}{6} = $$
And... now what? Can this even be solved? is it correct to say that there is no solution as $$\sqrt{3}$$ does not exist in Z7? Am I missing something?
You are correct that there is no solution, but not for the reason you're giving here. The quadratic equation only holds if we're working in $\mathbb R$ or $\mathbb C$ or something similar (like a subring of one of those). In the case of $\mathbb Z / 7\mathbb Z$, there are only seven elements to consider, so you can simply try plugging each of them into $3x^2 + x + 1$ and verify that none of them is a zero.