given a finite permutation group $G$ and a list of subgroups of $G$ list1. I want to compute the fixed points of $U$ acting on $G/V$ via right multiplication, where $U, V \in$ list1 (i.e. a submatrix of the table of marks of $G$ ).
If the groups are not too big, one can use $ fix_X( G/V ) = | \{ Y^g : g\in G, X \leq Y^g \}|\cdot [ N_G( Y ) : Y ]$, where $Y^g = g^{-1}Yg$ is the right conjugacy operation of $G$ on its subgroups. My implementation in GAP using this formula works fine.
In my opinion $ | \{ Y^g : g\in G, X \leq Y^g \}| = | \{ ^gX : g\in G, ^gX \leq Y \}| $ holds; $^gX = gXg^{-1}$ being the left conjugcy operation. But using this formula in my GAP code the results are wrong.
Before posting my code: Is there any mistake in the above equality?
The problem in your equality is the (dis-)counting of duplicates when looking at sets. Suppose e.g. that $Y=G$ and $X$ a non-normal subgroup. Then there is one conjugate of $Y$ but many of $X$, all contained in $Y$.
I would give the following two options a try -- not necessarily because they are fundamentally better, but they have tuned code behind them:
returns (a homomorphism) the (right) permutation action on the cosets of $V$ in $G$.You could calculate the image of $U$ and determine the number of fixed points. Alternatively, you could compute
and check how many double cosets have size $|V|$. (The best algorithm would probably be to use the double coset calculation and discard any intermediate results that correspond to orbits of length $>1$. However this would require modifying the code.