I am trying to find the number of homomorphisms $ \phi: _4 \to _4 $ (where $_4$ is the dihedral group with $2\cdot4=8$ elements).
I know that $ _4 $ is generated by the $6$ transpositions ${(12),(13),(23),(14),(24),(34)}$ and a homomorphism is determined by the generators. Moreover, ${\rm ord}(\phi(a)) $ must divide ${\rm ord}(a) $. Hence, the order of the image of any generator is either $2$ or $1$. There are $4$ elements of order $2$ in $D_4$. But here is where I get stuck.
How do these observations help me find the total number of such homomorphisms?
Before looking at the generators, first observe that some elements must map to the identity!
Because $D_4$ has no elements of order $3$, every $3$-cycle must map to the identity. Moreover, every $2$-$2$-cycle can be written as a product of $3$-cycles (e.g. $(12)(34) = (123)(234)$), so any product of transpositions must also go to $0$.
Next, the kernel of $\phi$ must be a normal subgroup of $S_4$. The only normal subgroups of $S_4$ are $S_4, A_4, V_4$ and the identity subgroup. We've just seen that $V_4 = \{1, (12)(34), (13)(24), (14)(23)\}$ has to map to the identity. Hence, by the first isomorphism theorem, any homomorphism $$\phi: S_4\to D_4$$ automatically factors through $S_4/V_4 \cong S_3$. Hence, it is completely equivalent to count the homomorphisms $S_3\to D_4$.
But now, $C_3\subset S_3$ maps to the identity, so a homomorphism $S_3\to D_4$ factors through $S_3/C_3\cong C_2$.
We are left with counting the number of homomorphisms $$C_2\to D_4,$$ or equivalently, the number of elements of order dividing $2$ in $D_4$.
Since the map $S_4 \to C_2$ is obtained by quotienting by $A_4$, we deduce that all maps $S_4\to D_4$ take all non-transpositions to the identity and all transpositions to an element of order $2$ in $D_4$.