Let $G,H$ be abelian groups. Under what conditions would the following holds true $\mathrm{Hom}(\mathrm{Hom}(G,H),H) \simeq G$?
For example, if $G$ is any abelian group and $H=\Bbb R/\Bbb Z$ the isomorphism holds true. This is well-known as The Pontryagin duality theorem. Is there any counterexample?
You're asking a very hard question to answer.
Let's work in the more restrictive assumption that $G$ is finitely generated, so $G=\mathbb{Z}^n\oplus t(G)$, with $t(G)$ the torsion part of $G$. Then $$\DeclareMathOperator{\Hom}{Hom} \Hom(G,H)\cong H^n\oplus\Hom(t(G),H) $$ Note that $\Hom(t(G),H)$ is torsion of finite exponent. Then $$ \Hom(\Hom(G,H),H)\cong \Hom(H^n\oplus\Hom(t(G),H),H)\cong \Hom(H^n,H)\oplus\Hom(\Hom(t(G),H),H) $$ The second summand is torsion, so the rank only comes from the first summand, which is $\Hom(H,H)^n$.
In particular the rank of $\Hom(H,H)$ cannot be $>1$. It must be $1$ if $G$ is not torsion, which means that $\Hom(H,H)/K$ is torsion, where $K$ is the image of the unique ring homomorphism $\mathbb{Z}\to\Hom(H,H)$.
If $G$ is not torsionfree, then $H$ has to be not torsionfree as well.
Note that there is no group $H$ so that $\Hom(\Hom(G,H),H)\cong G$ for every abelian group $G$. The case you mention of $\mathbb{R}/\mathbb{Z}$ doesn't work generally, but only for finite groups. I'll leave to you finding the reason. The Pontryagin dual of a group $G$ is $G^*=\Hom(G,\mathbb{R}/\mathbb{Z})$ endowed with the topology induced by the product topology, where $\mathbb{R}/\mathbb{Z}$ is identified with the circle. The the double dual is the group of continuous homomorphism $G^*\to\mathbb{R}/\mathbb{Z}$ and it is this “restricted” dual that's isomorphic to $G$.