According to the article on Wikipedia about finite differences, the forward and backward difference by $h$ of a function $f(x)$ divided by $h$ approach the derivative to order $O(h)$, i.e.
$$\frac {\Delta_h f(x)}{h} -f'(x)= \frac{f(x+h)-f(x)}{h} -f'(x) \in O(h)$$
$$\frac {\nabla_h f(x)}{h} -f'(x)= \frac{f(x)-f(x-h)}{h} -f'(x) \in O(h)$$
However, from Taylor's theorem, I'd expect this to be $O(h^2)$, which following the article, is only the case for the central difference.
$$\frac {\delta_h f(x)}{h} -f'(x)= \frac{f(x+h/2)-f(x-h/2)}{h} -f'(x) \in O(h^2)$$
Can anybody explain why? Or give an example of when the forward (/backward) difference only approach to order $O(h)$?
Taylor's theorem gives $$ f(x+h) = f(x) + h f'(x) + O(h^2) $$ (for $h \to 0$) and therefore "only" $$ \frac{f(x+h)-f(x)}{h} -f'(x) = O(h) $$
A simple example is $f(x) = x^2$, where $$ \frac{f(x+h)-f(x)}{h} -f'(x) = \frac{(x+h)^2-x^2}{h} - 2x = h $$
For central differences, the function $g(h) = f(x+h) - f(x-h)$ has the property that not only $g(0) = 0$ but also $g''(0) = 0$, therefore $$ f(x+ \frac h2) - f(x - \frac h2) = g(\frac h2) = \frac h2 g'(0) + O(h^3) = h f'(x) + O(h^3) \, . $$