The integers are, up to isomorphism, the unique infinite discrete abelian group that is isomorphic to all its non-trivial subgroups. This can be seen easily by Pontryagin Duality: It's equivalent to saying that $S^1$ is the only infinite compact abelian group isomorphic to all its proper quotients by closed subgroups. Indeed, if G is an infinite compact abelian group with such property, then if $\chi$ is any non trivial character of G, we get an isomorphism of topological groups $G/ker(\chi) \cong Im(\chi)$ (compacity of $G/ker(\chi)$ guarantees it's a homeomorphism). The property imposed on G implies that G is isomorphic to a compact subgroup of $S^1$, and since G is infinite, we get $G \cong S^1$.
One can also show, in a similar spirit, that the only infinite discrete abelian groups isomorphic to all their proper quotients are the Prüfer p-groups $\mathbb{Z}(p^{\infty})$, although this takes a bit more work. By Pontryagin duality, this implies that the only compact infinite abelian groups isomorphic to all their closed non-trivial subgroups are the p-adic integers $\mathbb{Z}_p$.
So we can look at the property P of a topological (Hausdorff) abelian group being isomorphic to all closed non-trivial subgroups, and if we restrict ourselves to the categories of infinite discrete and infinite compact abelian groups, we get two different answers in each case (up to isomorphism): For discrete groups, only the integers satisfy P, and for compact groups only the p-adic integers satisfy it.
At the same time, it's well known that the two mentioned categories are in correspondence by Pontryagin duality, so with that in mind, it almost feels like there is a "duality" between the integers and the family of groups of p-adic integers. Is there anything actually interesting happening from a categorical point of view at all, or am I overthinking it?
Edit: I just noticed that we don't even need to purposely restrict ourselves to compact or discrete groups. Indeed, if G is any locally compact abelian abelian group isomorphic to all its non-trivial closed subgroups, then it will in particular have to be monothetic. As such, it's known that it's either compact or isomorphic to the integers. So the weird role played by the integers and the p-adic integers technically extends to locally compact abelian groups.
You are describing similar properties between $\mathbf Z$ among discrete abelian groups and every $\mathbf Z_p$ among compact abelian groups.
All I want to do here is record some other ways they appear in analogous roles.
$S^1 \cong \mathbf R/\mathbf Z$ and $\mathbf Z(p^\infty) \cong \mathbf Q_p/\mathbf Z_p$ as topological groups: $\mathbf Z$ is discrete and cocompact in $\mathbf R$, while $\mathbf Z_p$ is compact and open (= codiscrete) in $\mathbf Q_p$, with $\mathbf R$ and each $\mathbf Q_p$ being self-dual groups.
In $\mathbf R^n$, a lattice can be defined concretely as the $\mathbf Z$-span of a basis and conceptually as a discrete cocompact subgroup. In $\mathbf Q_p^n$, a lattice can be defined concretely as the $\mathbf Z_p$-span of a basis and conceptually as a compact open subgroup.