Let $x$ be a real variable. It’s valid to say,
As $x \rightarrow \infty$, $\frac{1}{x} \to 0 $
since $$\lim_{x \to \infty} \frac{1}{x} = 0$$
But is it also valid to say,
As $\frac{1}{x} \to 0$, $x \to \infty $
Since
$$\lim_{\frac{1}{x} \to 0^+} x = \infty$$
is different from
$$\lim_{\frac{1}{x} \to 0^-} x = -\infty$$
The first statement is correct, i.e. as $x \to \infty$, we have $\frac1x \to 0$.
Conversely, if $\frac1x \to 0$ that does not imply that $x \to \infty$ since it could be $x \to -\infty$, as you indeed point out. You could, however, say that as $\frac1x \to 0$ we have $|x| \to \infty$, or as $\frac1x \to 0^+$, we have $x \to \infty$.