Error while calculating automorphism of non abelian group on Gap

Question

I am implementing the following code to calculate the automorphism of non-abelian group on Gap.

f := FreeGroup("b", "c");    # define free group f in two generators
b := f.1; c := f.2;          # call b and c the generators of f
rels := [b^4, (b*c)^2, c^4]; # define relators
G := f/rels;                 # define G

Now, when I run

aut := AutomorphismGroup(G);

to calculate the automorphism group of G, I get the following output and error:

#I  Coset table calculation failed -- trying with bigger table limit
#I  Coset table calculation failed -- trying with bigger table limit
#I  Coset table calculation failed -- trying with bigger table limit
#I  Coset table calculation failed -- trying with bigger table limit
Error, reached the pre-set memory limit
(change it with the -o command line option) in
  prev[2 * limit] := 2 * limit - 1; at /proc/cygdrive/C/gap4r8/lib/grpfp.gi:1201 called from
TCENUM.CosetTableFromGensAndRels( fgens, grels, fsgens
 ) at /proc/cygdrive/C/gap4r8/lib/grpfp.gi:1032 called from
CosetTableFromGensAndRels( fgens, grels, List( trial, UnderlyingElement )
 ) at /proc/cygdrive/C/gap4r8/lib/grpfp.gi:3699 called from
Attempt( trial ) at /proc/cygdrive/C/gap4r8/lib/grpfp.gi:3702 called from
Attempt( gens ) at /proc/cygdrive/C/gap4r8/lib/grpfp.gi:3724 called from
FinIndexCyclicSubgroupGenerator( G, infinity ) at /proc/cygdrive/C/gap4r8/lib/grpfp.gi:3773 called from
...  at line 5 of *stdin*
you can 'return;'
brk>

See also a screenshot of this output and error mesage.

Answer 1

What happens is that GAP tries to prove that your group is finite. This fails as your group is infinite.

GAP does not have any method to calculate automorphism groups of arbitrary infinite (finitely presented) groups, in fact I doubt that such an algorithm (can) exist.

The error you get therefore is not a bug, but a feature of the problem you ask.

Your best bet might be to look at the automorphism group of (increasingly larger) characteristic quotients to try to deduce a general pattern of the automorphism group in the hope to prove it later by hand.