The definition of first derivative is $$f'(x) = \lim_{y \to x} \frac{f(y) - f(x)}{y-x}.$$ Iterating, we have for the second derivative $$f''(x) = \lim_{y \to x} \frac{f'(y) - f'(x)}{y-x}.$$ Combining the two, we have $$f''(x) = \lim_{y \to x} \frac{1}{y-x} \left[\lim_{z \to y} \frac{f(z)-f(y)}{z-y} - \lim_{w \to x} \frac{f(w)-f(x)}{w-x}\right].$$ This would imply that we need to fix four points, $x,y,z$ and $w$ and then take the above limits where all of them tend to the point $x$.
However, from the geometric interpretation, we know that the second derivative describes the curvature (of the osculating circle) at a point $x$. But, to specify a circle, we need only three, and not four point.
Therefore, there should be a way of writing the expression for the second derivative which has only three, and not four values of the independent variable. How to get this expression?
Consider the alternative formulation $$f'(x)=\lim_{h\to 0}\frac{f(x+h)-f(x)}{h}.$$ Therefore $$f''(x)=\lim_{h\to0}\frac{1}{h}\left(\lim_{h'\to 0}\frac{f(x+h+h')-f(x+h)}{h'}-\lim_{h'\to 0}\frac{f(x+h')-f(x)}{h'}\right).$$ Reordering gives $$f''(x)=\lim_{(h,h')\to 0}\frac{f(x+h+h')-f(x+h)-f(x+h')+f(x)}{hh'}.$$ Assuming that this limit exists (of course, we need that $f$ is sufficiently smooth) we may just consider the case $h=h'$. Thus $$f''(x)=\lim_{h\to 0}\frac{f(x+2h)+f(x)-2f(x+h)}{h^2}.$$