Let $G$ be a finitely generated CAT$(0)$ group and $H$ a subgroup. If $H$ is quasiconvex then it is finitely generated, so we can immediately conclude that any non-finitely generated subgroup of $G$ is not quasiconvex. Let us assume then that $H$ is finitely generated, and suppose $H$ is given as a finite set of generators $X\subset G$.
Question: Is there an algorithm which takes as input the group $G$ (in some suitable format) and the set $X$ and outputs whether $H=\langle X\rangle$ is not quasiconvex?
I am specifically interested in the case of a right-angled Coxeter system $(W,S)$ which is CAT$(0)$ as seen by making it act on their Davis complex - a CAT$(0)$ cube complex. In [1], Pallavi Dani and Ivan Levcovitz give a procedure (standard completion sequence) which takes in a finite set $X\subset W$ and which terminates after finitely many steps if and only if $\langle X\rangle$ is quasiconvex, see Theorem 8.4. I want to know if there is an algorithm, ie a procedure which terminates after finitely many steps, which tells me that a subgroup is not quasiconvex.
I have searched on MSE and MO, as well as Googling the literature on the subject but found nothing. I know there are certain invariants which can help prove in certain cases that a subgroup is not quasiconvex (eg growth rate), but this does not amount to a generally applicable algorithm.
I have asked the question in the very general setting of CAT$(0)$ groups, but would be happy with an algorithm which works in a restricted setting, eg CAT$(-1)$ groups; groups which act on a CAT$(0)$ cube complex (compare Theorem 8.4 mentioned above with Theorem 5.1 in [2]); right-angled Coxeter groups, etc.
If it turns out that this problem is undecidable, that would also be interesting.
[1] Pallavi Dani and Ivan Levcovitz Subgroups of right-angled Coxeter groups via Stallings-like techniques, 2021.
[2] Michael Ben-Zvi, Robert Kropholler, and Rylee Alanza Lyman Folding-like techniques for CAT$(0)$ cube complexes, 2020.
Edit: The definition of quasiconvex subgroup I am using is as follows: a subgroup $H$ of $G$ is $M$-quasiconvex, for $M \ge 0$, if any geodesic path in the Cayley graph of $G$ with endpoints in $H$ lies in the $M$-neighborhood of $H$. $H$ is quasiconvex if it is $M$-quasiconvex for some $M$. Whether or not a subgroup is quasiconvex does not depend on which finite generating set of $G$ is used to construct the Cayley graph.