I have a question how is this derivation possible, and could somebody explain with sources and recommendations. I know how to do Taylor Series but this is new and foreign and I have not seen anything like this. I found this while I was programming some derivatives in Fortran.Numerical Differentiation of Continuous Functions: Higher Order Derivatives: Theory
Edit
I found the course pretty useful, but I am still wondering where there is a background for material being used.This is the course I saw
Edit
I hope the question is not lacking any details. If it is I have done Taylor series, and basic derivative definition using limits.
The relevant result here is Taylor's formula, not Taylor's series. Just look up books on finite differences and you'll find plenty of rigorous proofs of these results. In the case of $f''$ you can just write down Taylor's formula for $f(x+h)$ and $f(x-h)$ and get $$ f(x+h)=f(x)+f'(x) h + f''(x) \frac{h^2}{2}+f'''(x)\frac{h^3}{6} + \frac{f''''(\xi_1) h^4}{24}, \quad\xi_1 \in (x,x+h) $$
$$ f(x-h)=f(x)-f'(x) h + f''(x) \frac{h^2}{2}-f'''(x)\frac{h^3}{6} + \frac{f''''(\xi_2) h^4}{24}, \quad \xi_2 (x-h,x) $$
Adding up these equalities, \begin{equation} \label{eq1} f(x+h)+f(x-h)= 2f(x)+ h^2 f''(x) + \frac{h^4}{24}(f''''(\xi_1)+f''''(\xi_2)). \end{equation}
Noting that, $$ \min_{(\xi_2, \xi_1)} f'''' \leq \frac 12 (f''''(\xi_1)+f''''(\xi_2)) \leq \max_{(\xi_2, \xi_1)} f'''' $$
you can see that $\frac 12 (f''''(\xi_1)+f''''(\xi_2)) = f''''(\xi), \quad \xi \in (\xi_2, \xi_1) $ and obtain $$f(x+h)+f(x-h)= 2f(x)+ h^2 f''(x) + \frac{h^4}{12} f''''(\xi).$$
Finally, solving with respect to $f''(x)$, yields $$ f''(x) = \dfrac{f(x+h)-2f(x)+f(x-h)}{h^2}-\frac{h^2}{12}f''''(\xi). $$ The term $-\frac{h^2}{12}f''''(\xi)$ is the one mentioned in the video as $o(h^2)$.