Why Does Adding the nth Derivative Increase a Function Approximation's Accuracy?

Question

I am currently taking calculus 3: sequences and series, and we've just started learning about Maclaurin and Taylor Series. I understand the concept behind them -- of these polynomials and derivatives of polynomials.

However, I do not understand physically why, when we have a function approximation $g(x) \approx f(x)$, adding more and more derivatives of $f(x)$ increases the accuracy of $g(x)$ more and more.

If someone could point me to a resource or explain it in simple terms it would be much appreciated. Thank you!

Answer 1

Consider a polynomial function $P$ and let $x_0$ be a real number. Then $P(x_0)$ is the value that $P(x)$ takes at $x_0$ and, since polynomial functions are continuous, whe $x$ is close to $x_0$, then $P(x)$ is close to $P(x_0)$.

Now, consider the polynomial $P_1(x)$ of degree $1$ such that $P_1(x_0)=P(x_0)$ and that $P_1'(x_0)=P'(x_0)$. It turns out that $P_1(x)=P(x_0)+P'(x_0)(x-x_0)$. On the other hand, if you write $P(x)$ as polynomial in $x-x_0$ rather than in $x$, what you will get will be$$P(x)=P(x_0)+P'(x_0)(x-x_0)+\frac{P''(x_0)}{2!}(x-x_0)^2+\cdots+\frac{P^{(n)}(x_0)}{n!}(x-x_0)^2.$$So, $P_1(x)$ is the best first degree approximation of this function near $x_0$, in the sense that$$\lim_{x\to x_0}\frac{P(x)-P_1(x)}{(x-x_0)^2}\tag1$$exists; if in $(1)$, $P_1(x)$ is replaced by any other first degree polynomial, that limit will not exist.

Now, if $P_2(x)=P(x)=P(x_0)+P'(x_0)(x-x_0)+\frac{P''(x_0)}{2!}(x-x_0)^2$, then $P_2(x)$ is the only second degree polynomial such that the limit$$\lim_{x\to x_0}\frac{P(x)-P_2(x)}{(x-x_0)^3}$$exists, and so on.

And since most functions behave locally like polynomials, approximating such a function $f$ by$$f(x_0)+f'(x_0)(x-x_0)+\frac{f''(x_0)}{2!}(x-x_0)^2+\cdots+\frac{f^{(n)}(x_0)}{n!}(x-x_0)^2,$$will be, in general, a good idea.

Answer 2

It is not necessarily the case, actually, that adding more terms will increase accuracy. In fact, there is no guarantee that the error will go to zero at all when adding more terms, even if the function is infinitely differentiable (smooth). That is why being analytic is stronger property than being smooth. A good example of a non-analytic smooth function is

$$f(x)=\begin{cases}0,&x=0\\\exp\left(-\frac1{x^2}\right),&x\ne0\end{cases}$$

which has $0$ as the Taylor polynomial at every order. It is easy to see this is the case, since $f(x)$ goes to $0$ as $x\to0$ faster than any non-zero polynomial. Hence why it's Taylor expansion has $f(x)$ as the error, which does not improve when higher order terms are added on.

One can say that adding on more terms guarantees a tighter asymptotic behavior near the point of interest though. In the above example, with each successive term, we can argue that the error is $\mathcal O(x^n)$ for successively larger $n$ (which is true since the error is $\mathcal O(x^n)$ for all $n$).

On the other hand, if the next term is non-zero, then you can argue that the error for the previous Taylor expansion is, in fact, $\mathcal O(x^n)$, but not less, as was the case with the above example.