So in texts in algebraic geometry and sheaf cohomology I'm seeing sheaves of abelian groups most commonly mentioned. I'm wondering what is the reason for this, why not sheaves of commutative rings, or sets? Why is the abelian group so fundamental as the sheaf attachment?
Thank you for your help.
On
Just want to point out that commutative rings also don't form an abelian category (and hence sheaves of them don't either). On the other hand, while sheaves of sets don't figure into the usual homological machinery, they do of course appear in the Yoneda lemma, and give you one way to consider generalizations of schemes.
Specifically, the Yoneda lemma shows that the category of schemes can be embedded in the category of contravariant functors $\textbf{Schemes}\rightarrow\textbf{Sets}$. If one picks a "Grothendieck topology" on $\textbf{Schemes}$, it makes sense to speak of what it means for a contravariant functor $F : \textbf{Schemes}\rightarrow\textbf{Sets}$ to be a sheaf. It turns out that for every scheme $X$, the functor $T\mapsto \text{Hom}(T,X)$ is a sheaf for any reasonable choice of Grothendieck topology $\tau$ on $\textbf{Schemes}$, the Yoneda lemma gives a fully faithful embedding of the category of schemes into the category of sheaves on $\textbf{Schemes}$ (with respect to $\tau$).
Thus, this latter category of sheaves is an enlargement of the category of schemes. This ends up being quite useful when discussing the notion of quotients of schemes by group actions (often the quotient doesn't exist as a scheme, but does exist as a sheaf). This leads to the notion of algebraic spaces, which in turn leads to the notion of algbraic stacks (these are essentially sheaves taking values in $\textbf{Groupoids}$ instead of $\textbf{Sets}$), which are extremely useful in the study of moduli problems.
Let $X$ be a scheme. Then there exist natural functors from certain sheaf categories associated to $X$, to certain algebraic categories, always defined by taking global sections.
There are several choices we could make. For the category of sheaves, we can take the category of quasi-coherent modules, the category of all $\mathcal{O}_X$-modules, or the category of sheaves of abelian groups. Similarly, you could choose the algebraic category to be the category of abelian groups, or if $X$ is an $R$-scheme, then we might as well take the category of $R$-modules.
So what choices are we supposed to make? As it turns out, in this case it doesn't matter much.
Let's first look at the 'domain' category. As it turns out, it is best to not use the category of quasi-coherent modules, for two reasons. First, the existence of enough injectives for the category of quasi-coherent sheaves is quite hard; second, the category of quasi-coherent sheaves is not closed under pushforwards, unlike the category of sheaves of modules and the category of sheaves of abelian groups. This makes proving certain things trickier.
So there's two remaining 'domain' categories, namely the $\mathcal{O}_X$-modules and the sheaves of abelian groups. Here, too, the output will not change. I believe Hartshorne treats this in his book Algebraic Geometry, somewhere in the beginning of Chapter 3. It may well be that the proof he gives can be adapted to show the claimed result above that I failed to reference. Why then prefer one over the other? Maybe for whatever purposes you have one approach might be more convenient then the other but in the end it doesn't really matter.
(That said, you can make things work by working with quasi-coherent sheaves. The resulting sheaf cohomology should not change, but I do not have a reference for this. It might be that the proof technique Hartshorne uses to compare modules with abelian groups can be applied here too, but the details might get trickier.)
Next, let's look at the 'output' category. There's two choices (abelian groups or $R$-modules), and both are quite fashionable, as far as I know. Here, the only difference is that taking the output to be in the category of $R$-modules might retain some more information. You can 'forget' about the module structure and your output will be simply an abelian group. I guess it all depends on how much info you really need. Hartshorne works with cohomology of unbased schemes with outputs as abelian groups, but you could similarly define a cohomology functor for $R$-schemes with outputs as $R$-modules, and I'd bet that pretty much every result would carry over without problems.
EDIT: You could, of course, come up with yet more variants. You can consider the sheaf of sets on a scheme too if you wish, but, as is said in the comments, this category lacks the required structure to make things work. You could maybe come up with more exotic structures on sheaves too, and, who knows, there might be some wacky 'cohomology theory' associated to it, but I'm not sure if this leads to anything, let alone if it's interesting.