Let $G,\cdot,d$ be a group in a metric space
Let $\phi:G\to Y$ be an injective group isomorphism such that $Y\subsetneq G$ but satisfying $\lim_{n\to\infty}\phi^n(G)=G$. Here $\subsetneq$ indicates proper subset (rather than subgroup) and $n$ indicates composition.
This looks for all the world like a homeomorphism to me, but it's unclear to me whether having an inverse means a homeomorphism must be onto.
Can a homeomorphism not have $Y\subsetneq G$?
$\phi[G]$ is a subgroup of $Y$ (just group theory) (which is itself a proper subgroup of $G$, if I understand you correctly). So $\phi^n[G]$ is just going to be only smaller, as $\phi^2[G]=\phi[\phi[G]]\subseteq \phi[G]$ and so on by induction. So the supposed limit (however that limit is defined, their intersection is the only reasonable guess as we have a decreasing family of subgroups of $Y$) can only itself be some subgroup of $Y$ and never $G$. So the question is moot, IMO.