All of the Eilenberg-Steenrod axioms for homology can relatively easily be translated into the language of category theory. We can then replace the abelian groups with a general abelian category.
Has this been done before? If not what complications will probably arise doing this? Do more axioms need to be added in order to say anything useful?
I have never seen such a generalization, but of course I am not omniscient. I do not see any problems in replacing the category of abelian groups with a general abelian category, but I doubt that it will produce something new.
The Freyd-Mitchell embedding theorem says that for each (small) abelian category $A$ there exist a ring $R$ and a fully faithful and exact functor $F: A → R$-$Mod$ to the category of left $R$-modules. See https://en.wikipedia.org/wiki/Mitchell%27s_embedding_theorem. That is, $F$ translates the new kind of homology theory into a "standard" theory.