This is a bit of a curiosity that intrigues me. Let $p$ be a prime and consider the sum of reciprocals of squares divisible by $p$. This is just $$ \dfrac{1}{p^2}\sum_{n=1}^\infty \dfrac{1}{n^2} = \dfrac{\pi^2}{6p^2}. $$ Then the sum of the reciprocals of squares not divisible by $p$ is $$ S = \sum_{\substack{n=1\\ p \nmid n}}^\infty \dfrac{1}{n^2}= \sum_{n=1}^\infty\dfrac{1}{n^2} - \dfrac{\pi^2}{6p^2} = \dfrac{\pi^2}{6} \left( 1 - \dfrac{1}{p^2} \right). $$ Now here's a bit of trouble (for me at least). We can view the terms of $S$ as elements in the field $\mathbb{F}_p$. Indeed, there's no division by the characteristic of the field, $p$. Since the field is closed under addition, the addition of each term is an element of the field. Therefore the addition of every element in the sum gives an element in the field. However $\frac{\pi^2}{6} ( 1 - \frac{1}{p^2} )$ is clearly not an element of $\mathbb{F}_p$.
I think the problem lies in that the limit of the partial sums does not exist modulo $p$, at least in this case. However, are there instances where it makes sense to add an infinite number of elements in a finite field, say perhaps when the sequence of terms has finite period? Thanks
As @AviSteiner has notes, infinite sums are defined in terms of a convergence of partial sums. Discussion of convergence of sequences requires a topology. Furthermore, a convergent sequence is not guaranteed to converge to a unique point unless the topology is Hausdorff. Since I expect you would like to say that your infinite sum of points in the field converges to something unique (provided it converges at all), we should make the topology on our finite field Hausdorff. Incidentally, the only Hausdorff topology on a finite set is discrete. In the discrete topology, the only convergent sequences are constant after a while. This leads to a rather disappointing answer: in order to guarantee a unique solution to your series, you must have that all but finitely many terms in the series are the additive identity.