For a differentiable function $f:\mathbb R\to\mathbb R$, we have, by definition,
$$f'(x)=\lim_{h\to0} \frac{f(x+h)-f(x)}h.$$
Is there a similar construction for higher derivatives, i.e. is it possible to write $$f^{(n)}=\lim_{h\to0} \Delta_f^n(x,h),$$
where $\Delta_f^n(x,h)$ is a rational function depending only on $$h, f(x), f(x+h), f(x+2h),\dots, f(x+n h)$$ ?
My attempt: See my answer below.
Yes! The idea is to start off with $\Delta^1_f(x,h)\overset{\text{Def.}}=\frac{f(x+h)-f(x)}h$ and to then recursively define $$\Delta^{n+1}_f(x,h)\overset{\text{Def.}}=\frac{\Delta^n_f(x+h,h)-\Delta^n_f(x,h)}h.$$
This construction leads to the following closed form, as can be checked by induction: $$\Delta_f^{n}(x,h)=\frac{\sum_{k=0}^n f(x+ k h) \binom nk (-1)^{n-k}}{h^n}.$$
Now it only remains to show that $\lim_{h\to0} \Delta_f^{n}(x,h) = f^{(n)}(x)$ for any $n$-times differentiable function $f$.
To this end, I will use l'Hôpital. First, I will prove two Lemmas
Lemma 1. Let $n\in\mathbb N$. Then for any polynomial function $P:\mathbb R\to\mathbb R$ of degree $<n$, $$\sum_{k=0}^n (-1)^{n-k} P(k) \binom nk=0.$$
Proof. Without loss of generality one can assume that $P$ is of the form $P(x)=x^m$ for some natural number $m<n$.
By the binomial Theorem, $$(e^x-1)^n=\sum_{k=0}^n \binom nk (-1)^{n-k} e^{kx},$$
so that $$\left.\frac{\mathrm d^m}{\mathrm d^mx}\right|_{x=0}(e^x-1)^n=\sum_{k=0}^n\binom nk (-1)^{n-k} k^m.$$
But $$\left.\frac{\mathrm d^m}{\mathrm d^mx}\right|_{x=0}(e^x-1)^n=\left.(e^x-1)^{n-m}\right|_{x=0} \cdot \text{something} = 0.$$
This achieves a proof. $\square$
Lemma 2. If $x\in\mathbb R$ is a number, $n\in\mathbb N$ is an integer, and $f:\mathbb R\to\mathbb R$ is differentiable in a neighbourhood of $x$, then $$\lim_{h\to0}\Delta^n_f(x+h,h)-\Delta^n_f(x,h)=0.$$
Proof. We have, using l'Hôpital $n$ times (thanks to Lemma 1, we can apply l'Hôpital $n$ times), \begin{split} \lim_{h\to0}\Delta^n_f(x+h,h)-\Delta^n_f(x,h) &= \lim_{h\to0}\frac{\sum_{k=0}^n (-1)^{n-k} \big(f(x+(k+1)h)-f(x+k h)\big)}{h^n} \\ &= \lim_{h\to0}\frac{\sum_{k=0}^n (-1)^{n-k} \big((k+1)f(x+(k+1)h)-kf(x+k h)\big)}{n h^{n-1}} \\ &= \dots \\ &= \lim_{h\to0}\frac{\sum_{k=0}^n (-1)^{n-k} \big((k+1)^n f(x+(k+1)h)-k^n f(x+k h)\big)}{n!} \\ &= \frac{f(x)}{n!} \sum_{k=0}^n (-1)^{n-k} ((k+1)^n-k^n) \binom nk, \end{split} and the last sum is equal to $0$ according to Lemma 1. $\square$
Now we can finally prove the desired result:
Theorem. For any $f$ that is differentiable $n$ times in a neighbourhood of $x\in\mathbb R$, $$\bbox[15px,border:1px groove navy]{\lim_{h\to0}\Delta^n_f(x,h)=f^{(n)}(x).}$$
Proof. Once again I use l'Hôpital: \begin{split}\lim_{h\to0} \Delta_f^{n}(x,h)&\overset{\text{H}}=\lim_{h\to0}\frac{\sum_{k=0}^n k\cdot f'(x+ k h) \binom nk (-1)^{n-k}}{n h^{n-1}} \\&\overset{k=0 \text{ term is }0}=\lim_{h\to0}\frac{\sum_{k=1}^n f'(x+ k h) \overbrace{\frac kn\binom nk}^{=\binom{n-1}{k-1}} (-1)^{n-k}}{h^{n-1}} \\&\overset{j\overset{\text{Def.}}=k-1}=\lim_{h\to0}\frac{\sum_{j=0}^{n-1} f'(x+ (j+1) h) \binom{n-1}j (-1)^{(n-1)-j}}{h^{n-1}} \\ &=\lim_{h\to0} \Delta^{n-1}_{f'}(x+h,h) \\ &\overset{\text{Lemma 2}}=\lim_{h\to0} \Delta^{n-1}_{f'}(x,h). \end{split}
Since $f'$ is differentiable $n-1$ times in a neighbourhood of $x$, this shows that if the Theorem is true for any $n-1$, then it is also true for $n$. Together with the fact that the Theorem is true by definition for $n=1$, a standard inductive argument shows that the Theorem is also true for any $n\in\mathbb N$.