When we define the semidirect product for $N \lhd G$ with $H < G$. We assumed $N$ is normal because that makes $f: N \times H \to G$, $f(n,h) = nh$ an isomorphism when we assume $f$ is a bijection. However, I don't know why can't we assume $H$ is normal and define semidirect product differently, namely, $(n_{1},h_{1}) \ast (n_{2},h_{2}) = (n_{1}n_{2},n_{2}h_{1}n_{2}^{-1}h_{2})$, or similarly it generalize to other homomorphism $\mu: N\to Aut(H)$.
If this is a valid construction, how does it relate to the original construction? Or in other words, if $H$ and $N$ are normal subgroups, I guess there is a natural isomorphism between $H \rtimes N$ vs $N \rtimes H$? By $(a,b) \mapsto (b,a)$?
I have done some search online but did not find much information, and I'm not sure my guess is correct. I'm totally a beginner in algebra and I would appreciate it if you can help me understand this simple question.
Let's start with the internal semidirect product, since that informs the construction.
Let $G$ be a group, and assume that:
Then every element $x\in G$ can be written uniquely as $x=nh$ with $n\in N$ and $h\in H$. We can write out how multiplication works, since given $x=nh$ and $y=n'h'$, we have $$xy = (nh)(n'h') = (nh)(n'(h^{-1}h)h') = \Bigl( n (hn'h^{-1})\Bigr)\Bigl(hh'\Bigr).$$ This works since $hn'h^{-1}\in N$, by the normality of $N$.
We can also consider that conjugation by $h$ induces a homomorphism $\varphi\colon H\to\mathrm{Aut}(N)$, where $h\in H$ maps to $\varphi_h$, with $\varphi_h(n) = hnh^{-1}$. Then our multiplication rule becomes: $$xy = (nh)(n'h') = (n\varphi_h(n'))(hh').$$ This suggests the construction of the (external) semidirect product of $N$ by $H$ (via $\varphi$): given any homomorphism $\varphi\colon H\to\mathrm{Aut}(N)$, we define a product on the set of all pairs $(n,h)\in N\times H$ by $$(n,h)\cdot (n',h') = (n\varphi_h(n'), hh'),$$ where $\varphi_h = \varphi(h)\in\mathrm{Aut}(N)$. One then proves that this is indeed a group, that $\mathcal{N}=\{(n,1)\in N\times H\mid n\in N\}$ is a normal subgroup that is isomorphic to $N$; that $\mathcal{H}=\{(1,h)\in N\times H\mid h\in H\}$ is a subgroup isomorphic to $H$; and we have $$(1,h)(n,1)(1,h)^{-1} = (\varphi_h(n),1),$$ so that conjugation by $h\in\mathcal{H}$ realizes $\varphi_h$. We denote this group by $N\rtimes_{\varphi}H$, with the notation suggesting that (i) the underlying set is $N\times H$; (ii) the subgroup $N$ will be normal; and (iii) $\varphi$ is part of the required information. This group satisfies:
Now, the thing is, going back to the original group $G$, since $NH$ is a group, it follows that $NH=HN$. So while every element $x\in G$ can be written uniquely as $x=nh$, it is also true that for every $x\in G$ there exist a unique expression of $x$ as $x=hn$ with $h\in H$ and $n\in N$ (not necessarily the same ones as in the first expression). Had we started with this expression, we would have the multiplication rule given as follows: if $x=hn$ and $y=h'n'$, then $$xy = (hn)(h'n') = hh'((h')^{-1}nh')n'.$$ If we let $\psi_h$ be "conjugation by $h$", then now we have $$(hn)(h'n') = hh'\psi_{(h')^{-1}}(n)n'.$$ This suggests that we could perform the entire construction using $H\times N$ as our underlying set. Indeed, letting $\psi\colon H\to\mathrm{Aut}(N)$, then we define a group with underlying set $H\times N$, and multiplication given by $$(h,n)\cdot (h',n') = (hh', \psi_{(h')^{-1}}(n)n').$$ Then one shows that this is indeed a group; that $\mathcal{N}=\{(1,n)\in H\times N\mid n\in N\}$ is a subgroup isomorphic to $N$, that $\mathcal{H}=\{(h,1)\in H\times N\mid h\in H\}$ is a subgroup isomorphic to $H$. And that conjugation of elements of $\mathcal{N}$ by elements of $\mathcal{H}$ is $$(h,1)^{-1}(1,n)(h,1) = (1,\psi_{h^{-1}}(n)).$$ If we denote this group by $H{}_{\psi}\!\ltimes N$, then we have:
So there was nothing intrinsically important about using $N\times H$ as our underlying set: we could have used $H\times N$. Note in addition that the two constructions are isomorphic if $\psi=\phi$. The key is again going back to the "internal" case: if $x=nh$, then $x = h(h^{-1}nh)$ is how to write it as an element of $H$ times an element of $N$.
So define $\Phi\colon N\rtimes_{\phi}H\to H{}_{\phi}\!\ltimes N$ by $$\Phi(n,h) = (h,\phi_{h^{-1}}(n)).$$ I claim this is an isomorphism. Indeed, to check it is a homomorphism, note that $$\begin{align*} \Phi\Bigl( (n,h)(n',h')\Bigr) &= \Phi(n\phi_h(n'),hh')\\ &= \Bigl(hh',\phi_{(hh')^{-1}}(n\phi_h(n'))\Bigr)\\ &= \Bigl(hh',\phi_{(h')^{-1}h^{-1}}(n)\phi_{(h')^{-1}h^{-1}} (\phi_h(n'))\Bigr)\\ &= \Bigl(hh',\phi_{(h')^{-1}}(\phi_{h^{-1}}(n))\phi_{(h')^{-1}}(\phi_{h^{-1}} (\phi_h(n')))\Bigr)\\ &= \Bigl(hh',\phi_{(h')^{-1}}(\phi_{h^{-1}}(n))\phi_{(h')^{-1}}(n')\Bigr)\\ &= \Bigl(hh',\phi_{(h')^{-1}}\bigl(\phi_{h^{-1}}(n)n'\bigr)\Bigr).\\ \Phi(n,h)\Phi(n',h') &= \Bigl(h,\phi_{h^{-1}}(n)\Bigr)\Bigl(h',\phi_{(h')^{-1}}(n')\Bigr)\\ &= \Bigl(hh',\phi_{(h')^{-1}}\bigl(\phi_{h^{-1}}(n)\bigr)\phi_{(h')^{-1}}(n')\Bigr)\\ &= \Bigl( hh', \phi_{(h')^{-1}}\bigl( \phi_{h^{-1}}(n)n'\bigr)\Bigr). \end{align*}$$ So $\Phi$ is a group homomorphism. It is one-to-one, since $\Phi(n,h)=(h,\phi_{h^{-1}}(n))=(1,1)$ if and only if $h=1$ and $1=\phi_{h^{-1}}(n) =\phi_{1}(n) = n$. And it is surjective, since given $(h,n)\in H\times N$, we have $$\Phi(\phi_h(n),h) = (h,\phi_{h^{-1}}(\phi_h(n))) = (h,\phi_{h^{-1}h}(n)) = (h,n).$$ Thus, $\Phi$ is an isomorphism.
So there is no difference between $N\rtimes_{\phi}H$ and $H{}_{\phi}\ltimes N$, except for how you write things: you get isomorphic groups. Some people prefer the $N$ coordinate first, some people prefer the $N$ coordinate second. We usually denote it $N\rtimes_{\phi} H$ in both cases, however, because we usually use $\triangleleft$ for the normality symbol (and write $N\triangleleft G$); we could just as easily write $G\triangleright N$ all the time, and then it would be more natural to write $H{}_{\phi}\!\ltimes N$, but we don't.
There's also nothing sacred about letting $N$ be the normal subgroup and $H$ be the subgroup acting on $N$; we could just as well say that $H$ is normal and $N$ acts on it and is not necessarily normal... except that we try to use visual cues to help understanding, and $N$ is normally used to denote a Normal subgroup.
So to avoid that cueing, say we have group $H$ and $K$, and morphisms $\phi\colon H\to\mathrm{Aut}(K)$, and $\psi\colon K\to\mathrm{Aut}(H)$. We can then construct $K\rtimes_{\phi}H$ (which we could also construct as $H{}_{\phi}\!\ltimes K$), and we can also construct $K{}_{\psi}\!\ltimes H$ (which we could also construct as $H\rtimes_{\psi}K$). In general, we don't expect the two groups, $K\rtimes_{\phi}H$ and $K{}_{\psi}\!\ltimes H$ to be isomorphic, even though they are both constructed on the underlying set $K\times H$. They could be isomorphic, but that will likely just be happenstance.
And more, $K\rtimes_{\phi}H$ is isomorphic to $K{}_{\psi}\!\ltimes H$ via the map that sends $(k,h)$ to $(k,h)$ if and only if $\phi$ and $\psi$ are both the trivial map and these semidirect product are actually direct products. Indeed, if $\phi$ and $\psi$ are both trivial then both semidirect products are equal and are direct products. Conversely, if the map is an isomorphism, then the image of $\mathcal{K}=\{(k,1)\mid k\in K\}$ is normal in $K{}_{\psi}\!\ltimes H$, as it $\mathcal{H}=\{(1,h)\mid h\in H\}$ by construction. So we are in the situation where we have two normal subgroup $\mathcal{K}$ and $\mathcal{N}$ of a group $G$, with $G=\mathcal{NK}$ and $\mathcal{N}\cap\mathcal{K}=\{e\}$. In this situation, it is well known that $nk=kn$ for all $k\in\mathcal{K}$ and $n\in\mathcal{N}$, so we actually have that $$(k,n) = (k,1)(1,n) = (1,n)(k,1) = (k,\psi_{k^{-1}}(n))$$ for all $k\in K$ and $n\in N$, so $\psi_{k^{-1}}(n)=n$ for all $n$; thus $\psi(k^{-1})=\mathrm{id}_N$, and this holds for all $k$. This yields as well that $\phi$ is the trivial map.