If I know that a category $\mathcal{D}$ is equivalent to a category of sheaves $\mathbf{Sh}(\mathcal{C},J)$, and I fix the small category $\mathcal{C}$, does the Grothendieck topology $J$ have to be unique?
(I know that there can be different sites of definition of the Grothendieck topos $\mathcal{D}$, but I am not interested in that. I only want to know how much (if at all) $J$ can vary if somehow I know what $\mathcal{C}$ is.)
It does depend on what structure you are implicitly respecting. For example, if $\mathcal{C}$ is a subcategory, and you insist that the composite $\mathcal{C} \hookrightarrow \mathcal{D} \xrightarrow{\equiv} \mathbf{Sh}(C, J)$ is naturally isomorphic to the Yoneda embedding, then $J$ is indeed unique, since this determines a specific geometric embedding $\mathcal{D} \hookrightarrow \mathbf{PSh}(\mathcal{C})$, and equivalence classes of such are bijective with Grothendieck topologies on $\mathcal{C}$.
But if you really care about bare equivalences of categories, then no; $\mathbf{PSh}(\mathbf{1} + \mathbf{1})$ has two different subtoposes that are equivalent to $\mathbf{Set}$, and they correspond to different topologies on $\mathbf{1} + \mathbf{1}$.