I need to count the number of group homomorphisms $$\# \left\{ \phi : \mathbb{Z}/a\mathbb{Z} \rightarrow \mathbb{Z}/b\mathbb{Z} \ \middle| \ a,b \in \mathbb{N}, a,b > 1 \right\}$$
But how many combinations do apply for a given combination of $a,b$? It's at most $$\frac{\min(\#\mathbb{Z}/a\mathbb{Z},\#\mathbb{Z}/b\mathbb{Z})}{2} - 1$$ as the neutral element needs to be mapped to its counterpart, and for each element the inverse element need to be mapped to the targets inverse counterpart as well.
It would be easy for group isomorphisms, as it would require $a=b$, but what constraints do I need here? Can you help me to go on?
These are cyclic groups. A homomorphism is uniquely determined by where you map a generating element, so you only need to consider what that element can be mapped to.
In this case we need $\phi(x+y)=\phi(x)+\phi(y)$, and $\phi(0)=0$. The important thing to notice is that since $a\cdot 1=0$ (1 summed with itself $a$ times) we need that $\phi(a\cdot 1)=a\cdot \phi(1)=0$. This condition is necessary and sufficient to determine where a generator can be mapped. For example, there exists $\phi:\mathbb{Z}/3\to \mathbb{Z}/6$ with $\phi(1)=2$, since $3\cdot 2=0$ in $\mathbb{Z}/6$.