Lemma 10.26. Let $G$ be a group acting transitively on a set $Y$, and let $y_0 \in Y$. Then $$\mathrm{Aut}(Y) \cong N_G(G_0)/G_0,$$ where $G_0$ is the stabilizer of $y_0$.
$N_G(G_0)$ is the normalizer of $G_0$ and $\mathrm{Aut}(Y)$ is set of all $G-$ isomorphisms, $Y\rightarrow Y$ (bijective $G-$ maps $f(g\cdot y)=g\cdot f(y))$ .
I was trying to understand this lemma in the book of Rotman, but I am confused while understanding the well-defined part, mainly how they defined the map. Can anyone please explain how the map is well-defined?
Proof. Let $\varphi \in \mathrm{Aut}(Y)$. Since $G$ acts transitively on $Y$, there is $g \in G$ with $\varphi(y_0) = gy_0$. First, we show that $g \in N_G(G_0)$. If $h \in G_0$, then $hy_0 = y_0$ and $$gy_0 = \varphi(y_0) = \varphi(hy_0) = h\varphi(y_0) = hgy_0;$$ hence $y_0 = g^{-1}hgy_0$ and $g^{-1}hg \in G_0$, as desired. Second, if $\varphi(y_0) = gy_0 = g_1y_0$, then $g^{-1}g_1$ fixes $y_0$ and $gG_0 = g_1G_0$. Therefore the function $$\Gamma: \mathrm{Aut}(Y) \to N_G(G_0)/G_0$$ defined by $$\Gamma(\varphi) = g^{-1}G_0,$$ where $\varphi(y_0) = gy_0$, is a well-defined function.
To avoid this, I did it another way. So, I defined a map $h: N_G(G_0) \rightarrow \mathrm{Aut}(Y)$ by $g \mapsto \psi$ where $\psi(y_0) = gy_0$. Clearly the kernel is $G_0$, and by transitivity of the action it is surjective, so once it is a well defined homomorphism we can apply 1st isomorphism theorem to get the conclusion, but again I am getting trouble while showing that $h$ is a homomorphism. It would be helpful if anyone could help with this.
Fix $y_0$. Given $\varphi\in\mathrm{Aut}_G(Y)$, $\varphi(y_0)\in Y$. Since the action is transitive, there exists a $g\in G$ such that $gy_0=\varphi(y_0)$. Rotman wants to map it to a coset of $G_0$ in $N_G(G_0)$, namely to the coset $g^{-1}G_0$. But $g$ need not be unique. So he needs to verify two things for this function to make sense:
So, first, he shows that $g\in N_G(G_0)$. To do that, we need to show that if $h\in G_0$, then $g^{-1}hg\in G_0$. Indeed, we have that because $h\in G_0$ then $hy_0=y_0$; and because $gy_0 = \varphi(y_0)$, we have that $$\begin{align*} (g^{-1}hg)\cdot y_0 &= (g^{-1}h)\cdot(g\cdot y_0)\\ &=( g^{-1}h)\cdot\varphi(y_0)\\ &= g^{-1}\cdot (h\cdot\varphi(y_0))\\ &= g^{-1}\cdot\varphi(h\cdot y_0)\\ &= g^{-1}\cdot\varphi(y_0) &\text{(since }hy_0=y_0\text{)}\\ &= g^{-1}\cdot(gy_0) \\ &= (g^{-1}g)\cdot y_0\\ &= e\cdot y_0\\ &= y_0. \end{align*}$$ So $g^{-1}hg\in G_0$. Therefore, $g$ (and $g^{-1}$) normalize $G_0$. So they do lie in $N_G(G_0)$.
Now, to show the map to $N_G(G_0)/G_0$ does not depend on the choice of $g$, suppose that $g'$ also has the property that $g'y_0 = \varphi(y_0)=gy_0$. We need to show that $g^{-1}G_0 = g'^{-1}G_0$. This is equivalent to showing that $gG_0=g'G_0$, because we know that $g$ and $g'$ will normalize $G_0$; and this is equivalent to $G_0 = g^{-1}g'G_0$, or again equivalently, that $g^{-1}g'\in G_0$. That is, that $(g^{-1}g')y_0 = y_0$. Indeed, we have $$(g^{-1}g')y_0 = g^{-1}(g'y_0) = g^{-1}(gy_0) = (g^{-1}g)y_0 = y_0.$$ So $g^{-1}g'\in G_0$, and therefore, $gG_0 = g'G_0$. And since both $g$ and $g'$ normalize $G_0$ this means that $g^{-1}G_0=g'^{-1}G_0$. So we can map $\varphi$ to the coset $g^{-1}G_0$, and this map gives an element of $N_G(G_0)/G_0$, and the element does not depend on the choice of which $g$ satisfies $gy_0 = \varphi(y_0)$.
For example, consider $G=\langle (1,2,3,4),(2,4)\rangle\leq S_4$ acting on $Y=\{1,2,3,4\}$, and let $y_0 = 1$. The only element that fixes $1$ is $(2,4)$, so the stabilizer (which I will write $G_1$ since it fixes $1$) is $G_1=\{e,(2,4)\}$.; the normalizer of $G_1$ is $\{e, (2,4), (1,3)(2,4), (1,3)\}$.
What are the $G$-automorphisms of $Y$? There's the identity, of course. There is also the bijection that swaps $1$ and $3$ and swaps $2$ and $4$. It is not hard to verify that this is indeed a $G$-automorphism, since $\pi=(1,3)(2,4)\in Z(G)$, so the bijection $y\mapsto \pi(y)$ satisfies that $g(\pi(y)) = (g\pi)(y) = (\pi g)(y) = \pi(gy)$ for all $g\in G$ and $y\in Y$. It is not hard to check that these are in fact the only $G$-automorphisms of $Y$.
Now, the identity is mapped to the coset of the inverse of any element of $G$ that fixes $1$. The only elements that fix $1$ are $e$ and $(2,4)$, and they are both their own inverses and correspond to the same coset of $G_1 = \{e,(2,4)\}$. And the element $\pi$ should be mapped to the coset of the inverse of any element that sends $1$ to $3$. There are two elements that send $1$ to $3$ in $G$: there is $(1,3)$; and there is $(1,3)(2,4)$. Both are their own inverses and give the same coset modulo $G_1$.
What you are trying to do does not work for multiple reasons.
First, you have not actually defined an image of $g$. You said you are mapping it to $\psi$, "where $\psi(y_0)=gy_0$."
First... why is there such an automorphism? You didn't define a bijection from $Y$ to $Y$, you just said what you want $\psi$ to do on a single element of $Y$. So I don't know what $\psi$ is. What is it on any other $y\in Y$? Do we know that such an automorphism exists? How do I know there is an automorphism that does that for each $g\in N_G(G_0)$?
And is there just one $\psi$ that sends $y_0$ to $gy_0$? Or are there many? I don't know. If there are many, which one do you pick?
And note that you may not get transitivity of $N_G(G_0)$ on $Y$ (as in the example above, the action of $N_G(G_1)$ is not transitive on $\{1,2,3,4\}$), so your argument for surjectivity also doesn't quite work. You would need to show that for every $\psi\in\mathrm{Aut}(Y)$ there is a $g\in N_G(G_0)$ with $gy_0=\psi(y_0)$... which would essentially be what Rotman is doing in the first part of the argument.
And even if you resolve all those issues... you wouldn't get a homomorphism. Because if $g$ corresponds to $\psi$ and $h$ corresponds to $\phi$, then it is not true that $gh$ corresponds to $\psi\circ\phi$. Somewhat counterintuitively, it corresponds to $\phi\circ\psi$: because $$\phi(\psi(y_0)) = \phi(gy_0) = g\phi(y_0) = g(hy_0) = (gh)y_0.$$ So what you get is not a homomorphism, but an anti-homomorphism. That's precisely why Rotman is using the inverse!
Instead, let us assume that for every $g\in N_G(G_0)$ you can find a unique automorphism $\varphi_g$ such that $\varphi_g(y_0) = gy_0$. Then you should map $g$ not to the automorphism $\varphi_g$, but rather to the automorphism $\varphi_{g^{-1}}$! That is, you need to send $g$ to the unique (assuming such a thing exists) $\psi\in\mathrm{Aut}(G)$ such that $\psi(y_0) = g^{-1}y_0$, not the one that sends $y_0$ to $gy_0$. Indeed, let $g,h\in N_G(G_0)$. Then $gh$ would correspond to the automorphism that sends $y_0$ to $(gh)^{-1}y_0$. So to verify the assignment is a homomorphism, we need to verify that $\varphi_{g^{-1}}\circ\varphi_{h^{-1}}$ sends $y_0$ to $(gh)^{-1}y_0$. And indeed: $$\begin{align*} \varphi_{g^{-1}}\circ\varphi_{h^{-1}}(y_0) &= \varphi_{g^{-1}}(\varphi_{h^{-1}}(y_0))\\ &= \varphi_{g^{-1}}(h^{-1}y_0) \\ &= h^{-1}\varphi_{g^{-1}}(y_0)\\ &= h^{-1}(g^{-1}y_0)\\ &= (h^{-1}g^{-1})y_0\\ &= (gh)^{-1}y_0. \end{align*}$$ So this assignment would be a homomorphism... provided that you could prove that there is in fact a unique $\varphi_g\in \mathrm{Aut}(Y)$ with the property that $\varphi_g(y_0) = gy_0$ for every $g\in N_G(G_0)$. Which you haven't done. Your map is just not necessarily a morphism (unless $\mathrm{Aut}(Y)$ happens to be abelian).