What is the index of $A\ast(ba)B(ba)^{-1}$ in $A\ast B$?

Question

A group is called co-Hopfian if it is not isomorphic to a proper subgroup of itself, while it is called finitely co-Hopfian if it is not isomorphic to a proper finite index subgroup of itself. A co-Hopfian group is finitely co-Hopfian, but the converse is not true (as the examples below will illustrate).

Let $A$ and $B$ be non-trivial groups. As I learnt from this answer, the free product $A\ast B$ is not co-Hopfian, i.e. there is a proper subgroup of $A\ast B$ which is isomorphic to $A\ast B$, namely $A\ast(ba)B(ba)^{-1}$ where $a \in A$ and $b \in B$ are non-identity elements.

What is the index of $A\ast(ba)B(ba)^{-1}$ in $A\ast B$?

I suspect that the index is always infinite. This is the case if $A\ast B$ is finitely co-Hopfian. Here are two such examples:

1. If $A$ and $B$ are free groups on $m$ and $n$ generators respectively, then $A\ast B$ is a free group on $r := m + n$ generators. Since $A$ and $B$ are non-trivial, we have $m, n \geq 1$ and hence $r \geq 2$. An index $k$ subgroup of a free group on $r$ generators is a free group on $k(r - 1) + 1$ generators, see here for example. If $k > 1$, then $k(r - 1) + 1 > r$, so $A\ast B$ is finitely co-Hopfian.

2. If $A$ and $B$ are non-trivial finite groups and not both $\mathbb{Z}_2$, then the Euler characteristic of $A\ast B$ is $\chi(A\ast B) = \frac{1}{A} + \frac{1}{B} - 1 \leq \frac{1}{2} + \frac{1}{3} - 1 = -\frac{1}{6}$, so $\chi(A\ast B) \neq 0$. Since the Euler characteristic is multiplicative under finite index subgroups, we see that such a group is finitely co-Hopfian.

The previous argument doesn't apply to $\mathbb{Z}_2\ast\mathbb{Z}_2$ since $\chi(\mathbb{Z}_2\ast\mathbb{Z}_2) = \frac{1}{2} + \frac{1}{2} - 1 = 0$. In fact, the group $\mathbb{Z}_2\ast\mathbb{Z}_2$ is not finitely co-Hopfian as it contains an index two subgroup isomorphic to itself, namely $\mathbb{Z}_2\ast a\mathbb{Z}_2a^{-1}$. This still leaves open the possibility that $\mathbb{Z}_2\ast(ba)\mathbb{Z}_2(ba)^{-1}$ has infinite index though.

My only other observation is that $A\ast(ba)B(ba)^{-1}$ is not a normal subgroup of $A\ast B$: for $h = (ba)b(ba)^{-1} \in A\ast(ba)B(ba)^{-1}$, and $g = (ba)^{-1} \in A\ast B$, we have $ghg^{-1} = (ba)^{-1}(ba)b(ba)^{-1}(ba) = b \not\in A\ast(ba)B(ba)^{-1}$.

Answer 1

Converting comments to answers.

Let $H=A\ast(ba)B(ba)^{-1}\le A\ast B$ be your subgroup. If $|B|>2$, then there's an element $c\in B\setminus\{1,b\}$. The cosets \begin{equation*} H(ca), H(ca)^2,H(ca)^3,\cdots \end{equation*} are all distinct. Indeed, equality $H(ca)^n=H(ca)^m$ for $n>m$ would mean $(ca)^{n-m}\in H$, but this is impossible, as all words in $H$ begin with $b$ or $a$, not $c$. Thus $H$ is infinite index.

If instead $|A|>2$, we can conjugate to show $(ba)^{-1}H(ba)$ is infinite index via the same argument above. Then of course $H$ is infinite index as well.

The case when $|A|=|B|=2$, which is the infinite dihedral group, is the sole counterexample. This is because in this case $A\ast B$ has an infinite cyclic subgroup $K$, which is normal and of index $2$. Since $|H|>2$, it must have nontrivial intersection $H\cap K$. But all nontrivial subgroups of $K$ are finite index, and thus $H\cap K$ is finite index in $K$, hence in $A\ast B$.

Answer 2

As Steve D shows, the only case where $H = A\ast(ba)B(ba)^{-1}$ has finite index in $G = A\ast B$, is when $|A| = |B| = 2$. It is natural to ask what the index is in this case.

Following Steve D's argument, consider $K = \langle ab\rangle < G$ which is infinite cyclic and has index two. Note that $(ab)^3 \in H$ and hence $\langle(ab)^3\rangle \leq H\cap K$. Suppose there is $(ab)^q \in H\cap K$ with $(ab)^q \not\in \langle(ab)^3\rangle$. The latter condition is equivalent to $q \equiv 1, 2\bmod 3$, so it follows that either $ab \in H\cap K$ or $(ab)^2 \in H\cap K$. If $(ab)^2 \in H\cap K$, then so too is $ab = (ab)^3(ab)^{-2}$. Since $a \in H$, if $ab \in H\cap K \leq H$, then $b = a(ab) \in H$ which is a contradiction, so $\langle(ab)^3\rangle = H\cap K$.

Therefore $[G:H\cap K] = [G:K][K:H\cap K] = 2\cdot 3 = 6$, so $6 = [G:H\cap K] = [G:H][H : H\cap K]$. As $b \in G\setminus H$ and $babab \in H\setminus (H\cap K)$, we see that $[G: H] > 1$ and $[H: H\cap K] > 1$, so $[G : H] \in \{2, 3\}$. If $[G : H] = 2$, then $H$ would be normal in $G$, but this isn't the case (see the last paragraph of the question).

Conclusion: The index of $\mathbb{Z}_2\ast(ba)\mathbb{Z}_2(ba)^{-1}$ in $\mathbb{Z}_2\ast\mathbb{Z}_2$ is $3$.