Let $X$ be a compact in the Polish space (metric, complete, separable) and $G\subseteq X\times X$ is open. For $x\in X$ we define the section of $G$: $$ s(x) = \{y\in X|\langle x,y \rangle \in \bar{G}\}. $$
The set $A'\subseteq X$ is invariant if for all $x\in A'$ holds $s(x)\subset A'$. How to verify if there are non-empty invariant subsets of given compact $A$? Maybe there are known equivalent problems?
It will be even helpful in the case $X = [0,1]$.
This is reformulated and changed a little bit problem from my previous question: Self-complete set in square
The sets $X$ and $\bigcup_{x\in X} s(x)$ are always invariant.
For any $X$, if $G=X\times X$ then the only invariant sets are $X$ and $\emptyset$.
If $X=[0,1]$ and $G=\{(x,y); |x-y|<\varepsilon\}$ for some given $\varepsilon>0$ then the only invariants sets are $X$ and $\emptyset$. (A similar set $G$ works for $X=\mathbb R$.)
So without some additional assumptions you cannot avoid the situation that the only non-empty invariant subset is $X$.