If $f$ is a function, what polynomial is a good approximation of order $n$ for $f$ near $x=0$?
Here we say that $P$ is a good approximation of order $n$ for $f$ near $x=0$ when $E(x)$ approaches $0$ faster than $h(x)=x^n$, where $E(x)=f(x)-P(x)$.
So far I have discovered that better approximations are found when $n$ is large in $x^n$. However, I'm not sure how to tackle this problem. Any help on where to start would be appreciated.
We have a function $f(x)$, and we want to find a polynomial $P(x)$ such that $$ \frac{f(x) - P(x)}{x^n} \to 0$$ as $x \to 0$. Let's start small, and say that $n=1$. It's immediately clear that $P(x)$ only needs to be linear, as any terms higher than quadratic don't matter in the fraction. (And more generally, we only need to find $P$ a poly up to degree $n$).
Denote $P(x) = a_0 + a_1 x$. So we have $$ \frac{f(x) - a_0 - a_1 x}{x} \to 0.\tag{1}$$ If $a_0 \neq f(0)$, then as $x \to 0$ we see clearly that $(1)$ doesn't go to $0$. In fact, it diverges. So $a_0 = f(0)$. Now we are looking at $$ \frac{f(x) - f(0) - a_1 x}{x} \to 0.\tag{2}$$ There are two ways to proceed, based on what you feel comfortable with. The easier way to proceed is to use l'Hopital's rule (under the assumption that we choose $a_1$ in the right way to make the numerator go to $0$), which brings us to the limit $$ f'(x) - a_1 \to 0,$$ from which we conclude that $a_1 = f'(0)$.
Let's say we want to analyze $(2)$ without using l'Hopital's rule. Another way of proceeding is to use the mean value theorem. That is, we know from the mean value theorem that $$ f(x) - f(0) = f'(c) x,$$ where $c \in (0, x)$. As $x \to 0$, we see that $c \to 0$. So if $f$ is continuously differentiable, then $$ \frac{f(x) - f(0) - a_1 x}{x} \approx \frac{f'(c)x - a_1 x}{x} = f'(c) - a_1 \to f'(0) - a_1 = 0,$$ from which we again conclude that $f'(0) = a_1$. I wrote $\approx$ above, but it's not too hard to make this argument rigorous.
So we have totally finished the degree $1$ case. Let's go one higher, and consider $n = 2$. Now we want to find $P(x) = b_0 + b_1 x + b_2 x^2$. Looking at $$ \frac{f(x) - b_0 - b_1x + b_2 x^2}{x^2} \to 0,$$ we see that this necessarily means that $$ \frac{f(x) - b_0 - b_1 x - b_2 x^2}{x} \to 0$$ as well, so that we already know that $b_0 = f(0)$ and $b_1 = f'(0)$. We only are left to consider $b_2$.
Again, you can either apply l'Hopital's rule (twice) or the mean value theorem (twice). I find l'Hopital's rule easier to describe, as we immediately see that $$ \frac{f''(x) - 2b_2}{1} \to 0,$$ which means that $b_2 = \frac{1}{2} f''(0)$.
You can continue in this fashion. You will very quickly see a pattern develop, and you can prove that pattern inductively.
Aside --- As it happens, I've written more fully about how to carry out this process using only the Mean Value Theorem in a few different ways. For instance, you can use the Mean Value Theorem to prove Cauchy's Mean Value Theorem, which you can use to prove l'Hopital's rule, which we used here. Or you can use the Mean Value Theorem to prove a higher order Mean Value Theorem, which turns out to give you Taylor's Theorem with remainder. I gave an undergraduate talk and wrote these notes about it with these details and some more. [Disclaimer: link to my personal site].