Centralizer subgroup: $$C_G(A)=\{g\in G\mid gag^{-1}=a,\forall a\in A\}$$ Center subgroup: $$Z(G)=\{g\in G\mid ga=ag,\forall a\in G\}$$
Centralizer and center are subgroups of $G$. The centralizer takes a subset $A\in G$ and the center always uses the entire group $G$. So the centralizer becomes center when $A=G$ $$ C_G(G)= Z(G). $$
My questions are that:
Under what precise mathematical conditions that the normalizer $N_G(A)$ equal to a normal subgroup $N$ of $G$? When is $N_G(A)$ not equal to a normal subgroup $N$? See for instances.
Since the quotient group is the $G/N$, are there analogous concepts of quotient (not a subgroup) for $G/N_G(A)$, important for what properties or theorems?
In short, I am exhausting the concepts of
Centralizer : center subgroup ~ normalizer : normal subgroup ~ ?: quotient group
The normalizer works "the other way around" to what you may think it does. The normalizer of $A$ in $G$ is the largest subgroup of $G$ that contains $A$ and in which $A$ is normal.
That is, $N_G(A) =\{g\in G:gA=Ag\}$. By its very definition, $A$ is normal in $N_G(A)$. However, $N_G(A)$ may not be a normal subgroup of $G$, it seems you already have examples of this. I am not sure you should expect any "reasonably general" condition that guarantees a normalizer is normal, but rather, the other way around.
One way in which normalizers and centralizers interact with quotients is as follows: each element $g\in N_G(A)$ defines an automorphism of $A$ by conjugation, and hence there is a map $N_G(A) \longrightarrow \mathrm{Aut}(A)$ that assigns $g$ to the automorphism $a\longmapsto {}^ga$. By definition, the kernel of this map is the centralizer of $A$ in $G$, so one obtains the "NC lemma" that $C_G(A)$ is normal in $N_G(A)$ and that $N_G(A)/C_G(A)$ is isomorphic to a subgroup of $\mathrm{Aut}(A)$.