Normalizer of a subset of algebraic group

Question

Let $G$ be a linear algebraic group (I'm thinking of these as subsets of affine space over an algebraically closed field.). Define the normalizer of a subset $H$ to be $\{g \in G: gHg^{-1} \subseteq H\}$. Is this equal to $\{g \in G:gHg^{-1}=H\}$?

Answer 1

This holds for any finite group and is not true for infinite groups in general. I'll denote $n(G):=\{g \in G: gHg^{-1} \subseteq H\}$ and $N(G) := \{g \in G: gHg^{-1} = H\}$. Clearly, we always have $N(G) \subseteq n(G)$. Let's show that $n(G)\subseteq N(G)$ holds if $G$ is finite: If $g \in n(G)$, the map \begin{align} \varphi_{g} :\; &H \rightarrow H \\ &h \mapsto ghg^{-1} \end{align} is well-defined. It's also easily seen to be injective: $$\varphi_g (h) = \varphi_g(h') \Leftrightarrow ghg^{-1} = gh'g^{-1} \Leftrightarrow hg^{-1} = h'g^{-1} \Leftrightarrow h = h'$$ Because $H$ is a finite set, every injective mapping from $H$ to $H$ is surjective, which shows $\varphi_g(H) = H$ which is equivalent to $g \in N(G)$. If $G$ is infinite, this is no longer true in general, see this question for a counterexample.

Answer 2

The right setup in which you should consider this question is when $H$ is a closed algebraic subset (i.e. Zariski-closed subset of $G$). If there is no algebraic structure on $H$, you won't be able to exploit the algebraic structure of $G$ and end up doing plain group theory where counter-examples can arise for cardinality reasons.

If $H$ is a Zariski-closed subset, write $H = \bigcup_{i=1}^n H_i$ where the $H_i$ are the irreducible components of $H$. If $gHg^{-1} \subseteq H$, we have $$ gHg^{-1} = \bigcup_{i=1}^n gH_ig^{-1} $$ and since conjugation by $g$ is an isomorphism of varieties, $gH_ig^{-1} \subseteq H$ is an irreducible subvariety of $H$. Therefore, conjugation simply permutes the irreducible components of $H$ and we conclude that $gHg^{-1} = H$.

Hope that helps,