I have a big issue in understanding the real meaning of Big O notation.
Classical definition: $f(x) = O(g(x))$ as $x\rightarrow k$ if there exist $\delta, C > 0$ such that $f(x) \leq Cg(x)$ whenever $|x-k| < \delta$.
I understand that the inequality holds when $|x-k| < \delta$ so we have an upper bound in that interval for $f(x)$ function. However, as there is a constant in the inequality I do not understand how it can help us to control the growth of the function. For example, if $C = 10^{100}$ then the bound would be quite different from $C = 1$ so I do not really know how big the bound of my function is.
The problem is related to finite difference method with Taylor series, $f\in C^2$: $$\frac{f(x+h) - f(x)}{h} - f'(x) = O(h)$$ which means that as $h\rightarrow 0$ if there exist $\delta, C > 0$ such that $\frac{f(x+h) - f(x)}{h} - f'(x) \leq Cx$ whenever $h < \delta$. So, the error between the real derivative and approximation is bounded by $Cx$ but that bound is quite different if $C = 10^{100}$ or $1$. So, how Big O notation tells me the maximum error that I will have? That error could be as big as wanted making C big enough?
Thanks a lot!
Big-$O$ notation doesn't tell you anything about the maximum error you can have; it tells you about the asymptotic behavior of the function. It tells you that there is a constant $C$; for a great many purposes you don't need to have any specific value for $C$, you simply need to know it exists. Any particular $C$ that fits into the definition would put a bound on the maximum error you could have, but the big-$O$ doesn't give you any hints about what $C$'s fit.
e.g. if you want to show that, if $f(x) = x^2$, then $f'(x) = 2x$, then all you need to know is that
$$ 0 \leq \left|\frac{(x+h)^2 - x^2}{h} - 2x \right| < C h $$
for some constant $C$. It doesn't matter what value the constant has, just that some constant exists. Then you could take the limit to get
$$ \lim_{h \to 0} 0 \leq \lim_{h \to 0}\left|\frac{(x+h)^2 - x^2}{h} - 2x \right| \leq \lim_{h \to 0} Ch $$
and thus
$$ 0 \leq \left| \left( \lim_{h \to 0} \frac{(x+h)^2 - x^2}{h} \right) - 2x \right| \leq 0 $$
and
$$ \lim_{h \to 0} \frac{(x+h)^2 - x^2}{h} = 2x $$