how to calculate the automorphisms of a group that fix a subgroup

Question

I have a finite (polycyclic) group $G$ and a subgroup $H<G$.

How do I calculate the subgroup of $Aut(G)$ that fixes $H$ pointwise : $S = \{ a \in Aut(G) : \forall h \in H,a(h)=h\}$

It would be nice if this can be done in a straight forward way in GAP but a general approach is welcome.

Answer 1

Pick a generating set $\{h_1, \ldots, h_n\}$ of $H$. Note that $\alpha \in \operatorname{Aut}(G)$ fixes $H$ pointwise if and only if it fixes all of the $h_i$.

As $\operatorname{Aut}(G)$ is finite, you can check for every $\alpha \in \operatorname{Aut}(G)$ whether or not $\alpha(h_i) = h_i$ for all $i \in \{1,\ldots,n\}$. This is of course very inefficient, but it works.

A faster way would probably be to use existing orbit-stabiliser algorithms. For each $h_i$, calculate its stabiliser under the natural action of $\operatorname{Aut}(G)$ on $G$, and then take the intersection of these stabilisers.

A further improvement, suggested by @ahulpke in the comments, is to calculate the stabilisers iteratively.

An implementation in GAP (with some minor optimisations), could be

pointwiseFixed := function( G, H )
    local S, h;
    S := AutomorphismGroup( G );
    for h in SmallGeneratingSet( H ) do
        S := Stabiliser( S, h, \^ );
    od;
    return S;
end;

which can then be used like this:

gap> G := SmallGroup( 1536, 123456 );;
gap> H := Subgroup( G, [ G.5, G.8, G.9, G.10, G.2*G.7, G.3*G.7 ] );;
gap> K := pointwiseFixed( G, H );
<group>
gap> ForAll( K, aut -> ForAll( H, h -> h = h^aut ) );
true