I understand that, in big-o notation, when we say that a function $f$ is $O(x^2)$ we're basically saying that
$$|f(x)|\le M |x^2|$$ for some constant $M>0$ and for all $x>x_0$ for some $x_0$
Now, for little-o, if we say that $g(x)$ is $o(x^2)$, we're saying that:
$$|g(x)|\le \epsilon|x^2|$$
for all constant $\epsilon>0$ and for all $x>x_0$ for some $x_0$
Now, in this book, it visualizes $o(f(x)$ and $O(f(x)$ radically different:
I can't understand what exactly is a graph of an $O(h)$ mapping. Is it a class of graphs of functions, or whatever? Also, it says it must cross the origin in all 3 visualizations. Could somebody give me a better explanation, or a visualization with real functions?

But $O$ and $o$ are under-specified without a statement such as $x\to\infty$ or $x\to 0$.
For example: "$x^2=O(x)$ as $x\to 0$" is true; but "$x^2=O(x)$ as $x\to \infty$" is false.
The intepretation you quoted at the beginning is about $x\to \infty$. In some areas, like computer science, people tend to consider this situation exclusively, so "$x\to\infty$", is understood without saying.
But the book is talking about $x\to 0$, which is a different situation; we are considering $x$ near $0$, not large $x$.
That is, its graph is contained in some bowtie-shaped region like the one between the red lines.
where I have shown that in smaller neighborhoods of $0$, one can use flatter bowtie (green).