In D. J. S. Robinson's, A Course in the Theory of Groups it is written (on page 16):
"If $X$ is a nonempty subset of a group $G$, the normal closure of $X$ in $G$ is the intersection of all the normal subgroups of $G$ which contain $X$. This is a normal subgroup; it is denoted by $$ X^G. $$ Clearly $X^G$ is the smallest normal subgroup containing $X$ and it is easy to show that $X^G = \langle g^{-1} X g : g \in G \rangle$. [...] Dual to the normal closure is $X_G$, the normal interior or core of $X$ in $G$; this is defined to be the join of all the normal subgroups of $G$ that are contained in $X$, with the convention that $X_G = 1$ if there are no such subgroups. Again it is smple to prove that $H_G = \bigcap_{g \in G} g^{-1} H g$ for $H$ a subgroup."
As written it should also be $$ X^G = \bigcap_{N \unlhd G, X \subseteq N} N $$ and I guess $$ X_G = \langle N : N \subseteq X, N \unlhd G \rangle = \prod_{N \subseteq X, N \unlhd G} N $$ where $\prod$ denotes the internal product $AB = \{ ab : a \in A, b \in B$ (this equality should follow because for normal $N$ and $H \le G$ it is $\langle N, H \rangle = NH$).
Is this right? But then maybe if $X$ is just some set, then $X \subsetneq X_G$? Just in case $X$ is a subgroup $X_G$ is contained in $X$, by definition of $\langle \cdots \rangle$.
Are these formulae right?
Also with respect to what lattices are these operations defined, and in what sense are they dual, could this duality be made more formal? For example $X_G$ could not be defined just with respect to the normal subgroups contained in $X$, because $X_G$ might be outside of $X$ if $X$ is not a subgroup itself. (But because $NM$ is also normal for $N,M \unlhd G$, $X_G$ is always a normal subgroup).