Let $C$ be a category endowed with a Grothendieck topology $J$ of covering sieves, giving us a site $(C,J)$. We say $(C,J)$ is locally connected if for any object $c\in C$ all covering sieves $j\in J(c)$ of $c$ are connected as full subcategories of the slice category $C/c$. [1]
Recall that a category is connected when its groupoid completion is connected. Note also that they seem to require that the covering sieves are nonempty.
It seems that this condition is always satisfied when the site comes from a topological space. Given any sieve $j\in J(c)$ let $a \to c, b\to c$ be elements from $j\in J(c)$ we may simply take the fiber product (=intersection) $a\times_c b$ which clearly maps to $a,b$ in $C/c$. Hence it seems all sieves are always connected. It is, of course, possible that $a\times_c b$ is the empty set, but this is allowed.
Concretely, the Cantor space $2^{\mathbb{N}}$, considered as a site in the obvious way, should not be locally connected. I was not able to show this using the above definition. How to show this?
In the site corresponding to a topological space, most objects have the property that all covering sieves are connected, for the reason that you explained, but there's an exception: The empty set is an object that is covered by the empty sieve. (I don't mean the sieve $\{\varnothing\}$, which also covers $\varnothing$; I mean the sieve $\varnothing$.) And this sieve is not connected.
Why not? I ordinarily define a category $C$ to be connected if, whenever it is represented as the disjoint union of a family of categories, the family contains $C$ itself. But the empty category is the disjoint union of the empty family of categories, which doesn't contain the empty category.
The nLab defines connectedness as "$\text{Hom}(C,-)$ preserves finite coproducts." Again, that makes the empty category not connected.