Let $G$ be a matrix Lie group, i.e. a closed subgroup of $GL_n(\mathbb{C})$. Let $H$ be any subgroup of $G$ and
$$ N_G(H) := \{ A \in G \, \vert \, AHA^{-1} = H \} $$
the normalizer of this subgroup. If $H$ is itself closed in $G$ then it is easy to show that the normalizer is closed as well.
I was wondering if there is an example of a non-trivial normalizer, i.e. $N_G(H) \subsetneq G$, such that it is closed in $G$ even though $H$ is not.
Sorry, I initially misread your question. Here is an example. Consider $H< SO(2)$, an infinite cyclic subgroup; it is necessarily dense in $SO(2)$. I consider $SO(2)$ as a subgroup of $G=SO(3)$, fixing two points $p, q\in S^2$. The subgroup $SO(2)$ normalizes $H$. At the same time, the normalizer of $H$ in $G$ is a bit larger, it is the index 2 extension of $H$, consisting of rotations that preserve the subset $\{p, q\}$ (but not elementwise). Hence, this normalizer is closed in $G$. Thus, you get an example of a non-closed subgroup $H< G$ such that $N_G(H)$ is closed but different from $G$.