Consider the function $$ f(x)=\left\{\begin{array}{rll} 1+x^2 & \text{if} & x \,\,\text{rational} \\ -x^2 & \text{if} & x \,\,\text{irrational}\end{array}\right. $$ Then, for $x=0$, the limit $\lim_{h\to 0}\dfrac{f(h)-f(-h)}{2h}$ exists, although $f$ nowhere continuous.
Consider now the function $$ f(x)=\left\{\begin{array}{rll} 1 & \text{if} & x=0 \\ 0 & \text{if} & x\ne 0\end{array}\right. $$ Then the limit $\lim_{h\to 0}\dfrac{f(x+h)-f(x-h)}{2h}$ exists, for every $x$, although $f$ is not continuous at $x=0$. This example can be generalised, and obtain an $f$ which is discontinuous in countably many points (for example all the rationals), while the central difference converges.
Suppose now that limit $\lim_{h\to 0}\dfrac{f(x+h)-f(x-h)}{2h}$ exists for every $x$ is some open interval. Does this imply that $f$ is not differentiable in at most countably many points?
Let $$g(x,h) = \dfrac{f(x+h)-f(x-h)}{2h},\quad g_0(h) = g(x_0,h).$$ Easy to see that $$g_0(h)=0\ \text{if}\ x_0 = 0,$$ as for any another rational $x_0.$
So the function $g_0(h)=0$ is differentiable in the first case, within $g'_0 = 0.$
In the second case, there is a removable discontinuity of the derivative in the point $x=0$.
In the third case, it can exist the arbitrary (countable) quantity of the removable discontinuities or the gaps of the derivative.