I have two questions that are confusing me.
Why is the cyclic group generated by $g$ the smallest subgroup containing $g$?
If $g$ generates an infinite subgroup, why is it called cyclic? I mean, there is no power $m\gt 0$ so that $g^m=e$, the identity, so $g$ does not cycle.
Not sure that this reasoning answers the first question but here it is: if $g$ generates a subgroup $G$, then $e,g,g^{-1}$ are all in $G$, now we add elements in pairs $g^2,(g^2)^{-1},\dots$. This is the smallest subgroup containing $g$ for if another subgroup contains $g$, it must contain another element $f$ and so is larger than the former group.
I would like to summarize my answer, cause we have been debating a lot in the comment section and it has got a little blurry.
So first let's define $\langle g \rangle$. In my algebra course it's defined as
$\langle g \rangle=\lbrace g^s | s \in \mathbb{Z} \rbrace $.
A cyclic group is defined as a group $G,*$ for which there exists $g \in G$ such that $\langle g \rangle $. This is equivalent with the wikipedia definition: http://en.wikipedia.org/wiki/Cyclic_group
Now for the first question: Suppose there exists a subgroup $H$ which contains $g$. We now have to prove that $\langle g \rangle \subset H$. No problem: since $H $is a group, all $g^s$ with $s \in \mathbb{Z}$ are in $H$. Therefore $\langle g \rangle \subset H$.
For the second question: It seems to me that the existence of $m>0$ such that $g^m$=e is not a necessary condition to prove that $\langle g \rangle$ is cyclic. If you take as convention/notation that $g^0=e$, you can easily see that $\langle g \rangle$ is a group. And suppose there would be an $m>0$ such that $g^m=0$. Then $g^{m+1}=g$ and thus $g$ cannot generate a infinite group. That's just a plain contradiction with the fact that $g$ does generate a infinite group.