The two groups in question are $\phi:\mathbb{Z}_{12}\to\mathbb{Z}_5$ and $\phi:D_4\to S_3$.
For the first group, the answer is "no homomorphism," because there must be 5 cosets in $\mathbb{Z}_{12}$, but 5 does not divide 12.
However, for the second group, the answer is $$\phi=\begin{cases} \rho_0,\rho_1,\rho_2,\rho_3\to(1\,2)^0\\ \mu_1,\mu_2,\delta_1,\delta_2\to(1\,2)^1\end{cases}$$
But by the logic of the first part, wouldn't there not exist a homomorphism from $D_4$ to $S_3$, since $\vert D_4\vert=8$ and $\vert S_3\vert=6$? In general, how do I solve these types of problems?
This is a good observation; the correct criteria would be that if $G$ and $H$ are groups whose orders are coprime, then there are no non-trivial homomorphisms between them. Here, $5$ and $12$ are coprime, but $8$ and $6$ are not. One more specific thing you can note is that Lagrange's theorem guarantees that the size of the image of a homomorphism $G\rightarrow H$ divides both $|G|$ and $|H|$ (i.e. divides $\gcd(|G|,|H|)$). The second homomorphism you list, for instance, has an image of size $2$, which is exactly $\gcd(8,6)$.