I often see $\mathcal{O}(x)$ or $o(x)$ notations in evaluating limits, sums and integrals like in this answer on one of my questions, but I have no idea what they mean. I thought that these notations and their meaning are taught in real analysis books but after reading 2 of them I can say I was wrong.
Since I am a self learner and I don't what courses should I take or not for pure math I have no Idea from which field(area) these notations came from.
So I want to specifically ask for books that deals with them (It is more about finding out what is that branch (area) of mathematics that these notations came from than finding out what these notations mean. because I think that there are more useful thing in that branch(area) or books) It would be better if there book(s) are only pure math books.
Landau introduced little-oh notation in Handbuch der Lehre von der Verteilung der Primzahlen (1909) while trying to provide a similar concept to the big-oh notation from Bachman who introduced it in Analytische Zahlentheorie (1894). It is also worth mentioning Hardy's work too: Order's of infinity (1910); his notation is superior to Landau's but by the time Hardy introduced his, Landau's notation was already commonplace and Hardy's notation never got the attention it deserved.
A more modern reference would be Dieudonne's Infinitesimal Calculus, Hermann, Paris (1971), specifically Ch 3. "Asymptotic Developments." The whole book is excellent as it deals with many non-orthodox topics of calculus.
In a nutshell, $f = o(g)$ means that $|f/g| \to 0$ whereas $f = O(g)$ means $|f/g|$ is bounded near the point of approximation. Intuitively, these are functional limit versions of $<$ and $\leq,$ respectively. There are also version of $\approx$ and $=,$ namely $f = \Theta(g)$ which means, obviously $f = O(g)$ and $g = O(f),$ and $f \sim g$ which means $f/g \to 1.$