Let $\mathcal{L}=\{0,S,+, \cdot, <\}$ be the language of arithmetic and let $\mathcal{A}$ be the standard model of arithmetic.
For each prime $p$, let $\phi_p(v)$ be the formula saying $v$ is a multiple of $p$. Observe that for each set $S$ of prime numbers, $Th(\mathcal{A}) \cup \{ \phi_p(c) \}_{p \in S} \cup \{\neg \phi_p(c)\}_{p \notin S}$ (in the language obtained by adding a constant symbol to $\mathcal{L}$) is finitely satisfiable, hence must have a countable model which we will denote $\mathcal{A}_S$.
Let $\mathcal{B}_S$ denote the reduct of $\mathcal{A}_S$ to $\mathcal{L}$.
I am trying to prove or disprove the following claim:
For a given $S$, $\{\mathcal{B}_T: \mathcal{B}_T$ is isomorphic to $\mathcal{B}_S\}$ is countable.
Here's what I tried: Let $f: \mathcal{B}_S \to \mathcal{B}_T$ be a bijective embedding. I'm not 100% sure about this step, but intuitively it seems that there should be an $a \in \mathcal{B}_S$ such that $\mathcal{B}_S \models \phi_p[a]$ for every $p \in S$. [Just take $a \in c^{\mathcal{A}_S}$.] Therefore, $\mathcal{B}_T \models \phi_p[f(a)]$ for every $p \in S$.
We also know $\mathcal{A}_T \models \phi_p(c)$ iff $p \in T$. I was thinking of trying to combine this with the previous statement to try to prove that $S \subseteq T$, but I don't know how to do it formally or even if this argument works at all.
The key observation is that countability of the $\mathcal{A}_S$s is crucial. There is a single $\mathcal{M}\models Th(\mathcal{A})$ such that for each set of primes $S$ there is an element of $\mathcal{M}$ divisible by exactly the (standard) primes which are in $S$. This means that if we didn't require countability each $\mathcal{A}_S$ could be an expansion of $\mathcal{M}$ by a single constant, and then all the $\mathcal{B}_S$s would be isomorphic.
Of course such an $\mathcal{M}$ must be uncountable, and this points towards the solution to your question. Given $\mathcal{X}\models Th(\mathcal{A})$, let $$F(\mathcal{X})=\{S\subseteq\mathsf{PRIMES}: \exists x\in\mathcal{X}\forall i\in \mathbb{N}(p_i\vert x\iff i\in S)\}$$ (allowing the obvious abuses of notation). If $\mathcal{X}$ is countable, $F(\mathcal{X})$ must also be countable (why?). Now we clearly have $\mathcal{X}\cong \mathcal{Y}\implies F(\mathcal{X})=F(\mathcal{Y})$ and $S\in F(\mathcal{B}_S)$. Do you see why this implies that for each $T$ the set $\{S: \mathcal{B}_S \cong\mathcal{B}_T\}$ must be countable?
(Incidentally, $F(\mathcal{X})$ is near-as-makes-no-difference just the standard system of $\mathcal{X}$, which is a very useful notion!)