I just sat for my Algebra finals today and I was stumped by a question.
Let $(G, *)$ be a group and let $S$ be a subset of $G$. Denote
$$N = \{g \in G: gsg^{-1} \in S \ \mathrm{for \ all} \ s \in S \}.$$
If $G$ is a finite group, then it can be shown quite easily that $(N, *)$ is a subgroup of $G$. Indeed, if $g, g' \in N$, then $g'sg'^{-1} \in S$ and so, $(gg')s(gg')^{-1} = g(g'sg'^{-1})g^{-1} \in S$. Since $N$ is finite (as $G$ is finite), it follows that $N$ is indeed a subgroup of $G$.
However, in the case of an infinite group, is $(N, *)$ always a subgroup of $G$?
This stumped me for quite a while and I only managed to make the following deduction.
If a counterexample exists, then the counterexample must be derived from a non-abelian group. For if $G$ is abelian, then
$$g \in N \implies gsg^{-1} \in S \implies g^{-1}sg \in S \implies g^{-1} \in N$$ and combined with what was proven in the earlier part, we can conclude that $N$ is a subgroup of $G$.
After looking up online, there is a similar concept known as the normaliser that requires $gSg^{-1} = S$. However, with this weaker condition here, I am unable to proceed. Are there any counterexamples (or proof) to it?
If you take the Baumslag-Solitar group
$$B(1,2):= \langle a,b : bab^{-1} = a^2 \rangle$$
and let $S$ be the subgroup $S = \langle a^2 \rangle$ then you get a counter-example. Clearly
$$bSb^{-1} = \langle ba^2b^{-1} \rangle = \langle a^4 \rangle \subset S,$$
whereas
$$b^{-1}Sb = \langle b^{-1} a^2 b \rangle = \langle a \rangle.$$
To show that $bSb^{-1}$ is not in $S$, it suffices to show that $a \notin \langle a^2 \rangle$, which is certainly true if $a$ has infinite order. To see $a$ has infinite order, you can use the following linear representation of $B(1,2)$:
$$a \rightarrow \left( \begin{matrix} 1 & 1 \\ 0 & 1 \end{matrix} \right), \quad b \rightarrow \left( \begin{matrix} 2 & 0 \\ 0 & 1 \end{matrix} \right).$$
I think this might be the simplest counterexample where $S$ is actually a subgroup of $G$.