I know that GAP uses Todd-Coxeter algorithm to form the coset table of a group but there are multiple options while forming the table using this algorithm, so I was wondering if there is any way to access the exact algorithm GAP uses to for the command "LowIndexSubgroup(G,n)" .
I looked everywhere for this on GAP manual and github but could not find the exact complete algorithm.
LowIndexSubgroupsfor finitely presented groups uses the low-index algorithm, not the Todd-Coxeter algorithm. The implementation is essentially in the functionDoLowIndexSubgroupsFpGroupIterator(and functions called by it) inlib/grpfp.gi, with some subroutines (tracing) in the kernel for performance reasons. It closely follows the low index algorithm as described in Section 5.4 of Holt/Eick/O'Brien, Handbook of Computational Group Theory.