I am looking for references for the two facts found in the bullet points at the bottom of p. 386 of Diamond & Shurman's "A First Course in Modular Forms"
- For a field $k$ and a curve $X/k$ of genus $g$, if $M$ is coprime to the characteristic of $k$,
$$\operatorname{Pic}^0(X)[M] \cong \left(\mathbb Z/M\mathbb Z \right)^{2g}$$
- If a curve $X/\mathbb Q$ has good reduction at prime $p \nmid M$, then the reduction map is injective on $\operatorname{Pic}^0(X)[M]$.
At the request of the OP, I am going expand on my comments.
Let us first note that if $g$ has genus $0$ then this is basically all moot, since in this case $X_{\overline{k}}$ is $\mathbb{P}^1_{\overline{k}}$ so that $\mathrm{Pic}^0_X$ is trivial.
We then recall the following fact.
Note that you may take objections to my definition of the Picard functor on two counts:
But, as an answer:
Also, let us note the following corollary that is useful to answer your questions.
The proof is simple: the abelian variety over $\mathcal{O}$ modelling $\mathrm{Pic}^0_X$ is $\mathrm{Pic}^0_\mathcal{X}$ for $\mathcal{X}$ a smooth proper model of $X$!
But, truthfully, this explanation elides one small detail that's handled by the following lemma.
Proof: The fact that $\mathcal{X}_{\overline{s}}$ is connected can be seen as follows. Note that $\mathcal{X}$ itself is connected: if not then $\mathcal{X}=C_1\sqcup C_2$, so then $X=(C_1)_\eta\sqcup (C_2)_\eta$ ($\eta$ is the generic point of $\mathrm{Spec}(\mathcal{O})$), so since $X$ is connected we deduce that (WLOG) $(C_1)_\eta$ is empty. But, since $C_1$ is an open in $\mathcal{X}$, which is flat over $\mathcal{O}$, it's flat over $\mathcal{O}$ and so this is impossible (just think that locally on $C_1$ this means that it's of the form $\mathrm{Spec}(A)$ with $A_F=0$, but $\mathcal{O}\to A$ is flat, so this is impossible). But, since $\mathcal{X}\to\mathrm{Spec}(\mathcal{O})$ is smooth, and the target is normal, we deduce that $\mathcal{X}$ is normal, and thus since it's also connected it's integral (e.g. see this). The connectedness then follows from general principles related to the Theorem on Formal Functions (e.g. see [Illusie, Corollary 2.23]).
The fact about genus $g$ is then easy as it follows from the connectedness of the fibers and the constancy of Euler characteristic in flat families (see [Vakil, Theorem 24.7.1]). $\blacksquare$
From all of this, we see that your question really can be interpretted as follows:
Question: Let $A$ be an abelian variety over a field $k$ of dimension $g$.
The proof of 1. is not super easy, and definitely requires some legwork. I suggested reading [Conrad 2, Theorem 4.2.2]. Although, some of the ideas already appear in the explanation of 2. below.
To prove 2., note that the map $[N]\colon \mathscr{A}\to\mathscr{A}$ is finite flat (see [Conrad 2, Theorem 4.1.1]), so $\mathscr{A}[N]\to\mathrm{Spec}(\mathbb{Z}_{(p)})$ is finite flat (since it's the base change of $[N]$ along the identity section). So $\mathscr{A}[N]\to\mathrm{Spec}(\mathbb{Z}_{(p)})$ is a finite flat group scheme. In fact, by 1., we know that it's of order $N^{2g}$, which is invertible in $\mathbb{Z}_{(p)}$. So, by the disucssion in the first bullet of 'Finite Flat Group Schemes' section of [Blog] we deduce that $\mathscr{A}[N]\to\mathrm{Spec}(\mathbb{Z}_{(p)})$ is finite etale. Thus, also we have that $\mathscr{A}[N]_{\mathcal{O}}\to\mathrm{Spec}(\mathcal{O})$ is finite etale where $\mathcal{O}$ is the localization of $\overline{\mathbb{Z}}$ (the integral closure of $\mathbb{Z}$ in $\overline{\mathbb{Q}}$) at a prime above $p$.
Note though that any finite etale morphism $f\colon X\to Y$ is split by a finite etale cover of $Y$. Indeed, one can take a Galois cover $\widetilde{X}\to Y$ of $f$ (see e.g. see Tag 0BN2) dominating $X\to Y$, and then check that $X\times_Y \widetilde{X}\to\widetilde{X}$ is split.
But, note that $\text{Spec}(\mathcal{O})$ has no non-trivial (connected) finite etale covers. Indeed, by Tag 0BQL such a finite etale cover would be the normalization of $\mathrm{Spec}(\mathcal{O})$ in a finite extension of $\mathrm{Frac}(\mathcal{O})=\overline{\mathbb{Q}}$, of which there are no non-trivial ones.
So, $\mathscr{A}[N]_{\mathcal{O}}\to\mathrm{Spec}(\mathcal{O})$ becomes split on a finite etale cover of $\mathrm{Spec}(\mathcal{O})$, but such covers are disjoint unions of isomorphisms, so $\mathscr{A}[N]_{\mathcal{O}}\to\mathrm{Spec}(\mathcal{O})$ must already be split. This means that, in particular, it represents the constant sheaf on $\mathrm{Spec}(\mathcal{O})$ so that for any $\mathcal{O}$-scheme $T$ one has that $\mathscr{A}[N]_{\mathcal{O}}(T)=(\mathscr{A}[N](\mathcal{O}))^{\#(\pi_0(T))}$ (at least for non-degenerate schemes, see this)--but note that since $\mathscr{A}[N](\mathcal{O})=A[N](\mathbb{Q})$ we know from the first part that this group is just $(\mathbb{Z}/N\mathbb{Z})^{2g}$. But, in particular this means that if $T\to T'$ is a morphism of $\mathcal{O}$-schemes, both of which are connected, then the induced map $\mathscr{A}[N]_\mathcal{O}(T')\to\mathscr{A}[N]_\mathcal{O}(T)$ is a bijection.
Apply this for the diagram
$$\mathrm{Spec}(\overline{\mathbb{Q}})\to\mathrm{Spec}(\mathcal{O})\leftarrow\mathrm{Spec}(\overline{\mathbb{F}_p})$$
to see that the maps in the diagram
$$\mathscr{A}[N](\overline{\mathbb{Q}})\leftarrow \mathscr{A}[N](\mathcal{O})\to \mathscr{A}[N](\overline{\mathbb{F}_p})$$
are isomorphisms. Since the reduction map is defined as the composition of the inverse of the left map with the right map, the claim follows.
This map is also equivariant for the action of $D$, the decomposition of group in $\mathrm{Gal}(\overline{\mathbb{Q}}/\mathbb{Q})$ corresponding to our chosen prime (by construction). Here we are having $D$ act on $\mathscr{A}[N](\overline{\mathbb{F}_p})$ via its quotient $D\to\mathrm{Gal}(\overline{\mathbb{F}_p}/\mathbb{F}_p)$.
In particular, let's fix an extension $L$ of $\mathbb{Q}$. Note that our choice of a prime of $\overline{\mathbb{Z}}$ lying over $p$ induces such a choice for $\mathcal{O}_L$ thus giving rise to a residue field $\ell$ and a decomposition group $D_L\subseteq D$. We then see that we get an isomorphism
$$A[N](\overline{\mathbb{Q}})^{D_L}\xrightarrow{\approx}\mathscr{A}[N](\overline{\mathbb{F}_p})^{\mathrm{Gal}(\overline{\mathbb{F}_p}/\ell)}=\mathscr{A}[N](\ell).$$
Of course, this source group is, in general, smaller than the group $A[N](L)=A[N](\overline{\mathbb{Q}})^{\mathrm{Gal}(\overline{\mathbb{Q}}/L)},$ and so one only gets an injection
$$A[N](L)\to \mathscr{A}[N](\ell)$$
which is not necessarily a surjection. This non-surjectivity is related to the fact that $L$ is not Henselian (with respect to the valuation determined by our lift of $p$)--indeed, if that were true then there would be no difference between $\mathrm{Gal}(\overline{\mathbb{Q}}/L)$ and $D_L$. Antoher way to see this, is that if we had the Henselian property then $A[N](L)=\mathscr{A}[N](\mathcal{O}_L)\to \mathscr{A}[N](\ell)$ would be surjective by Hensel's lemma.
References:
[Blog] https://ayoucis.wordpress.com/2015/08/10/p-divisible-groups-formal-groups-and-the-serre-tate-theorem/
[Conrad] http://virtualmath1.stanford.edu/~conrad/248BPage/handouts/pic.pdf
[Conrad 2] http://virtualmath1.stanford.edu/~conrad/249CS15Page/handouts/abvarnotes.pdf
[Illusie] http://staff.ustc.edu.cn/~yiouyang/Illusie.pdf
[Kleinman] https://arxiv.org/pdf/math/0504020.pdf
[Vakil] http://math.stanford.edu/~vakil/216blog/FOAGnov1817public.pdf