A group $(G,\circ)$ is finite if and only if it has finitely many subgroups.
Lemma: A group $(G,\circ)$ can be written as a union of cyclic subgroups.
Proof:
$$G=\bigcup_{g \in G}\left\{g \right\}$$
Every $g$ is in the cyclic group generated by itself,$\forall g \in G: g \in \langle g \rangle$ and hence $\forall g \in G: \left\{g \right\} \subseteq\langle g \rangle$,follows: $$G=\bigcup_{g \in G}\left\{g \right\} \subseteq \bigcup_{g \in G}\langle g \rangle \subseteq G$$
$\Longrightarrow$ If $G$ is finite then the set of all subsets of $G$ is a finite set, a subgroup is a subset with extra structures which means the set of subgroups of $G$ is finite.
$\Longleftarrow$ If $G$ has an element $g$ of infinite order, then the cyclic group generated by $\langle g \rangle$ is an infinite cyclic group and isomorphic to $\mathbb Z$ and $n \mathbb Z$ is a subgroup of $\mathbb Z$ for every positive integer $n$, there are infinitely many such subgroups, so $\langle g \rangle$ has infinitely many subgroups, from which we conclude that $G$ has infinitely many subgroups,a contradiction.
From the lemma we know that $G$ can be written as the union of cyclic subgroups, in the previous section it's shown that every cyclic subgroup of $G$ is finite and by the assumption, there are finitely many such subgroups, so $G$ is a finite union of finite cyclic subgroups and hence is a finite group.
The problem is that in all of the proofs one uses the fact that an infinite cyclic group is isomorphic to the set of integers, but I think it's enough to show that there is bijection $\varphi: \mathbb Z \to \langle g \rangle$ such that $ k \mapsto g^k$, and from here we conclude that that $\langle g \rangle$ has the same number of elements as $\mathbb Z$.
But that is what we mean when we say $\langle g\rangle$ is isomorphic to the integers. Your bijection is an isomorphism.
Also, note that in the context of group theory, we often want to use, as much as possible, group theoretic notions (a similar statement is true in general). Bijections between groups are, in general, not very useful in group theory, but isomorphisms are. So we phrase theorems and proofs, as much as possible, by using isomorphisms rather than bijections. A proof is easier to read when the number of notions to keep track of is smaller.