Let $M$ be a manifold equipped with an affine connection $\nabla$. By standard theorems of differential geometry we have convex normal coordinate balls around every point in which geodesics are axis. For every normal ball $N_p \subset M$ We have an obvious involution given by the reflection $s_p:\{x_1,x_2,\dots,x_n\} \mapsto \{-x_1,-x_2,\dots,-x_n\}$ where $x_j$ are the normal coordinates.
Definition: $M$ Is locally symmetric iff the $s_p$ is an affine transformation for all $p \in M$.
Which means $ds_p (\nabla_X Y)= \nabla_{ds_p X} {ds_pY}$ for every pair of vector fields $X,Y$ defined in a neighborhood of $p \in M$.
It's hard for me imagine a situation where this won't hold. For one at the point $p \in M$ nothing bad could happen because $ds_p |_p=-Id$ so any violation of this must happen at a nearby point. For connections with torsion I can somehow imagine how things could go wrong but for symmetric connection I'm absolutely intuition-less for how such a thing is possible.
What's the simplest example of a torsion free affine manifold which isn't locally symmetric?
So, the vast majority of manifolds are not locally symmetric. The most obvious example that comes to mind is the standard torus in 3-space. This has positive curvature just to the left of the topmost circle, and negative curvature just to the right, and hence there cannot possibly be a geodesic symmetry at any of the points on that top circle. Similarly with more or less anything! The set of symmetric spaces has been completely classified, and (compact) locally symmetric spaces are quotients of simply connected symmetric spaces by some discrete group of isometries - most manifolds cannot possibly be on this list (since that list is countable!)
But maybe what you'd really like to see is a specific, closed manifold that cannot possibly support a single locally symmetric Riemannian structure. (I'm not thinking about the more general question of whether there's an affine, torsion-free, locally symmetric connection.)
So note that a compact, locally symmetric manifold's universal cover is complete and locally symmetric. By the classification theorem, the universal cover must either be flat, have nonnegative sectional curvature, or have nonpositive sectional curvature. So we just need to cook up a compact manifold that can never have nonnegative sectional curvature nor nonpositive sectional curvature. There are quite a few of these. Suppose $M$ is a manifold of dimension $n$.
First, nonpositive sectional curvature implies that the universal cover of $M$ is contractible: this is the Cartan-Hadamard theorem.
Second, nonnegative sectional curvature implies that the Ricci curvature is nonnegative (because the Ricci curvature $\text{Ric}(\xi,\xi)$ is $(n-1)$ times the average value of the sectional curvature over planes that contain $\xi$ when $\xi$ is unit length - frustratingly, I can't find this written down in my copy of Petersen, but one can prove it by hand - it's more-or-less definition pushing.) Now Bochner proved that an $n$-manifold with nonnegative Ricci curvature has $b_1(M) \leq n$. (The proof, more or less, goes by using the Weitzenbock formula to see that every harmonic 1-form is parallel, hence determined by its value on some specific point, and hence there's at most an $n$-dimensional space of them.)
Now let's put these together. We want a manifold whose universal cover is not contractible and with $b_1(M) > n$. The simplest example that comes to mind is $\#_4 (S^2 \times S^1)$, the connected sum of four copies of $S^2 \times S^1)$. The universal cover of this has infinitely-generated $\pi_2$ (so is very much not aspherical), and clearly $b_1(M) = 4>3$. If desired, you could come up with irreducible examples (once that are not connected sums of other 3-manifolds). There are a lot, a lot of 3-manifolds already that are neither aspherical nor have $b_1(M) \leq 3$. So these all give manifolds that are not locally symmetric with respect to any metric.