"$f$ is asymptotically greater than $g$"

Question

What's the notation for ranking functions in increasing asymptotic order? I saw a notation with $\prec$ earlier, but now I can't find it. Would the following notation be correct?

$$\mathcal{O}(n)\prec \mathcal{O}(n\log n)$$

Answer 1

Donald Knuth, Oren Patashnik, and Ronald Graham in their textbook "Concrete Mathematics" (chapter 9) really use this notation for functions (not for sets of functions): $$f(n) \prec g(n) \iff \lim_{n \to \infty}\frac{f(n)}{g(n)} = 0. \tag{9.3}$$ This notation was introduced by Paul du Bois-Reymond in 1871. Also, as noted in the comments, you can write the same relation as $f(n) = o(g(n))$ meaning that $f(n) \in o(g(n))$. However $f(n) \prec g(n) \iff g(n) \succ f(n)$, but relation $\color{red}{o(g(n)) = f(n)}$ is invalid. Swapping $f$ and $g$ leads to $g(n) = \omega(f(n))$ or $g(n) \in \omega(f(n))$.