Consider the following situation; you can skip to the end for the actual question. I have a relation $b = f(r)$, and let's say that I know that $f$ is analytic and invertible in a neighborhood of $r=0$ (I don't know if analicity is necessary but let's go with it), and that its series expansion starts at second order, with coefficient equal to 1:
$$b = r^2 + \mathcal{O}(r^3)$$
I would like to invert this, to get $r$ as a function of $b$ to leading order and also to get the behavior of the next term. Inverting just the first part is easy, so what I want to do is to fill in the following equation:
$$r = \sqrt{b} + \mathcal{O}(\text{?})$$
I could do this is in a very non rigorous way like this: write $r = \sqrt{b} + cb^n$, where $n>1/2$ and we don't care about higher order terms. I can truncate the power series of $b$ to $b$ to $b = r^2 + Ar^3 + Br^4$ (higher order terms won't be necessary). Substituting the expansion of $r$ into the one for $b$, we can throw out the terms that are of higher order than all the others. This leaves
$$0 = b^{n+\frac12} + c b^{2n} + \frac{A}{c} b^{\frac32} + 3A b^{n+1} + \cdots.$$
The only way these terms can cancel is to take $n=1$, so we get
$$r = \sqrt{b} + \mathcal{O}(b)$$
But there's a problem: the constants don't match! If I choose $c$ to cancel the $b^{\frac32}$ terms then I can't cancel the $b^2$ ones, and vice-versa. I don't think I made any mistakes in the calculation, but even if I did adn everything works out, I still have the following
Question: How can I (more-or-less) rigorously invert a big-oh expression? To use my example again: If I have $b = r^2 + \mathcal{O}(r^3)$ and want to get $r = \sqrt{b} + \mathcal{O}(\text{?})$, how do I find out what $\text{?}$ is, and how do I prove it?
An elementary approach is to "bootstrap" your approximations. Start with a basic one: from $b \sim r^2$ we get $b^{3/2} \sim r^3$ (assuming $b,r \geq 0$). Then $O(r^3) = O(b^{3/2})$, so
$$ b = r^2 + O(r^3) $$
becomes
$$ r^2 = b + O(b^{3/2}). $$
Then, taking the square root,
$$ r = \sqrt{b+O(b^{3/2})} = \sqrt{b} \sqrt{1 + O(\sqrt{b})} = \sqrt{b}\left[1+O(\sqrt{b})\right] = \sqrt{b} + O(b) $$
by the binomial theorem.
So we've taken a rough approximation $b \sim r^2$ and turned it into a finer approximation $r = \sqrt{b} + O(b)$.