I am reading this:
My question is how to get the first formula "has an error of order of $O(h)$", and the second formula "has error terms on order of $O(h^2)$".
I tried the taylor series expansion for the first case (could someone let me know if it is correct my expansion):
$f(x+h) = f(x) +\frac{f^{'}(x)}{1!}(x+h-x) + \frac{f^{''}(x)}{2!}(x+h-x)^2 + ...$, so I then subtract f(x) from both sides and then divides both sides by h, but my expression becomes this:
$\frac{f(x+h) -f(x)}{h} = \frac{f^{'}(x)}{1!} + \frac{f^{''}(x)}{2!}h + ... $ , so how is this being order of $O(h)$?
similarly how is the second formula being order of $O(h^2)$?
Note the order of the error for the derivative is the expansion minus the derivative. Thus, in your line for $\frac{f(x+h)-f(x)}{h}$, when you subtract $f'(x)$ from your right hand side, you get all terms having a factor of $h$. Since the terms with higher powers of $h$ go to $0$ faster than those with lower powers, the order becomes the one with the lowest power of $h$, i.e., $\frac{f''(x)h}{2}$, which has a power of $1$ for the $h$, so it's an $O(h)$ error term.
In general, you have the following expansions:
$$f(x + h) = f(x) + (f'(x))(h) + \frac{(f''(x))(h^2)}{2} + \frac{(f'''(x))(h^3)}{6} + \ldots \tag{1}\label{eq1A}$$
$$f(x) = f(x) \tag{2}\label{eq2A}$$
$$f(x - h) = f(x) + (f'(x))(-h) + \frac{(f''(x))((-h)^2)}{2} + \frac{(f'''(x))((-h)^3)}{6} + \ldots \tag{3}\label{eq3A}$$
Thus, \eqref{eq1A} minus \eqref{eq2A}, and dividing by $h$, gives
$$\frac{f(x + h) - f(x)}{h} = f'(x) + \frac{(f''(x))(h)}{2} + \frac{(f'''(x))(h^2)}{6} + \ldots \tag{4}\label{eq4A}$$
and \eqref{eq1A} minus \eqref{eq3A}, and dividing by $2h$, gives
$$\frac{f(x + h) - f(x - h)}{2h} = f'(x) + \frac{(f'''(x))(h^2)}{6} + \ldots \tag{5}\label{eq5A}$$
Since $h^2 = (-h)^2$, the $f''(x)$ terms cancel, so there's no $h$ term compared to \eqref{eq4A}.
Thus, the lowest order in terms of powers of $h$ error terms for computing $f'(x)$ in \eqref{eq4A} has a power of $1$, i.e., it's $O(h)$, while in \eqref{eq5A} it has a power of $2$, i.e., it's $O(h^2)$.