Let $G=GL(2,\mathbb{Z_p})$ where $p$ is prime. Let $$H=\left\{\begin{bmatrix}1&b\\0&1\end{bmatrix}:b\in\mathbb{Z_p}\right\}$$ We notice that $H<G$. Also, we notice that the normalizer is $N_G(H)=\left\{\begin{bmatrix}a&b\\0&d\end{bmatrix}:ad\neq 0\right\}$.
Now let $A=\{xHx^{-1}:x\in G\}$ denote the set of conjugates of $H$. Also, let $G$ act on $A$ by conjugation, or in other words $g \cdot xHx^{-1}=gxHx^{-1}g^{-1}$ for all $g\in G$. For all $g\in G$, define $\phi_g:A\to A$ by $\phi_g(a)=g\cdot a$. Finally define $\Phi:G\to Sym_A$ by $\Phi(g)=\phi_g$. We want to find $Ker(\Phi)$. We have that $$Ker(\Phi)=\{g\in G:\phi_g=id_A\}=\{g\in G:\phi_g(xHx^{-1})=xHx^{-1}, \forall xHx^{-1}\in A\} \\=\{g\in G: gxHx^{-1}g^{-1}=xHx^{-1},\forall xHx^{-1}\in A\}$$.
I would like to know how I should proceed from here to calculating the kernel.