This is Proposition 5.1 in Introduction to Topological Manifolds by Lee
I understand the proof up until the point where he mentions that "because boundedness of $f^{-1}$ implies $F(x) \to 0$ as $x \to 0$"
But $f^{-1}[\mathbb{S}^{n-1}] \subseteq \mathbb{R}^n \not\subseteq \mathbb{R}$ so we can't use the Extreme Value Theorem here which is what gives boundedness of a function in the most general case (I think).
This is what the author means I think. I think that the author means that there exists a function $g : \mathbb{S}^{n-1} \to \mathbb{R}$ defined by $g(x) = d(0, f^{-1}(x)) = |f^{-1}(x)|$ which is continuous and thus $g[\mathbb{S}^{n-1}] \subseteq \mathbb{R}$ is compact so $g$ is bounded (by the extreme value theorem).
Hence $\sup(g[\mathbb{S}^{n-1}])$ exists which I'll call $\alpha$. Now pick $x \in \overline{\mathbb{B}^{n}}$, then we have $|x|\alpha \to 0$ as $x \to 0$ $\implies$ $|x| \cdot \left|f^{-1}\left(\frac{x}{|x|}\right)\right| \to 0$ as $x\to 0$ $\implies$ $|x| \cdot f^{-1}\left(\frac{x}{|x|}\right)\to 0$ as $x\to 0$ for $x \in \overline{\mathbb{B}^{n}}$ $\implies F(x) \to 0$ as $x \to 0$.
Is that what the author means? If my analysis above is a bit sloppy then please let me know so I can correct it.
What I'm basically trying to ask is, how can I rigorously show $|x|f^{-1}\left(\frac{x}{|x|}\right) \to 0$ as $x \to 0$ for $x \in \overline{\mathbb{B}^{n-1}}$?
The sphere is compact, $f^{-1}$ is continuous, therefore $f^{-1}(\Bbb S^{n-1})$ is compact in $\Bbb R^n$, therefore bounded.
There is no reason to limit yourself to $\Bbb R$-valued functions to apply this argument, what you call the Extreme Value Theorem is just a corollary of the fact that a continuous mapping maps a compact to a compact, and that compacts in $\Bbb R^n$ are exactly the closed bounded subsets.
Edit:
The fact that $f^{-1}(\Bbb S^{n-1})$ is bounded means literally that there is a positive constant $C$ such that $$0\leq \|f^{-1}(x)\|\leq C\qquad \forall x\in \Bbb S^{n-1}$$ where $\|\cdot\|$ is any fixed norm on $\Bbb R^n$.
Therefore $$0\leq \|x\|\cdot \left\lvert\left\lvert f^{-1}\left(\frac{x}{|x|}\right)\right|\right|\leq C\|x\|\qquad \forall x\in \Bbb R^{n}\setminus\{0\}$$
$$0\leq \|F(x)\|\leq C\|x\|\qquad$$
Conclude using the squeeze theorem.