Here is the exercise from Aluffi's Algebra.
Let $G=N \rtimes_\theta H$ be a semidirect product, and let $K$ be the subgroup of $G$ corresponding to $\ker(\theta) \subseteq H$. Prove that $K$ is the kernel of the action of $G$ on the set $G/H$ of left cosets of $H$.
Let $(n, h)$ be an element of the kernel of the action of G.
Then, for any $(n', h')$, we should have $(n, h)•(n', h')=(n', h'')$ (Since we consider the kernel of the action of $G$ on the left cosets of $H$, It doesn't need to consider how the choosen element behaves on $H$ is.)
and then it is that actually $(n, h)•(n', h')=(nhn'{h}^{-1}, h'')=(n', h'')$ for any $(n', h')$.
Then $n$ should be $1_N$ (consider $H$ of the left cosets of $H$). So here we can see $h$ conjugates every $n$ in $N$ in the following way $hn{h}^{-1}=n$, which is precisely the element of $\ker(\theta)$.
And the converse implication is straightforward.
Is this right? Can you give me more direct proof? I feel like I did too much calculation which is not necessarily needed.
Any comment will be appreciated. Tha nk you in advance.
If I understand your attempt correctly, you seem to assume that $\theta_h$ is $hn'h^{-1}$. This isn't necessarily the case. Keep in mind that $N$ and $H$ are arbitrary groups and the latter expression might not even make sense.
To prove the direction you're interested in, first note that any element of $G/H$ can be represented by $(x, 1)H$ for some $x \in N$. This simplifies the following computation a bit.
Next, let $(n, h)$ be an element of the kernel of the action of $G$ on $G/H$. For all $x \in N$, we have $$(n, h)(x, 1)H = (x, 1)H.$$
Thus, $$(x^{-1}, 1)(n \theta_h(x), h) \in H.$$
It follows that $x^{-1}n\theta_h(x) = 1$, and therefore $\theta_h(x) = n^{-1}x$. Since $\theta_h$ is an automorphism of $N$, this forces $n = 1$ and $\theta_h = 1$ as desired.