The question is actually a bit broader than this, but it is a good starting point. Given two Dihedral groups $D_4$ and $ D_3$ we wish to construct a nontrivial homomorphism $f:{ D_4\to\ D_3}$. Is there a way to do this without explicitly writing the groups as $$D_4=\{e_{D4},c,c^2,c^3,b,bc,bc^2,bc^3\} $$ and $$D_3=\{e_{D3},k,k^2,s,sk,sk^2\}$$
and then trying different functions out. I'm trying to(or would like to) find a way to prove the existence or nonexistence of such a homomorphism, without explicity findig it.
Is there a general way of determening the existence of a homomorphism between two groups without explicitly finding it?
$D_4$ has a subgroup $H$ of index $2$ which must be normal (explicitly, one example is given by rotations, but this also follows from general theory), the quotient $D_4/H$ is cyclic of order $2$. If we take any element in $D_3$ of order two and map the generator of $D_4/H$ to that element, we get a non-trivial homomorphism $D_4/H \to D_3$. Composition with the quotient map $D_4 \to D_4/H \to D_3$ gives us a non-trivial homomorphism from $D_4$ to $D_3$.
In general, if you're interested in homomorphism $G \to H$, then as @HagenvonEitzen mentions in the comments, it is a good idea to investigate the normal subgroups of $G$.
This can help you both ways: If you have a normal subgroup $K$, then any homomorphism $G/K \to H$ also gives rise to a homomorphism $G \to H$, as I used above. But $G/K$ may have a much simpler structure than $G$.
On the other hand, for any homomorphism $f: G \to H$, $\operatorname{ker}(f)$ is a normal subgroup of $G$, so if you know the normal subgroups of $G$, this can put some serious constraints on possible homomorphisms. The most extreme case is when $G$ is simple: then any homomorphism is either trivial or injective. This property is often exploited in proofs.
As a different example, for $n \geq 5$, the only normal subgroups of $S_n$ are the trivial group, $A_n$ and $S_n$ itself. This implies that if $G$ is any group of odd order and $|G| < n!$, then there is no non-trivial homomorphism $S_n \to G$.
If you know about group presentations, then the problem of finding homomorphisms can be recast this way: assuming you have a presentation $G= \langle S \mid R \rangle$, then finding a homomorphism $G \to H$ is equivalent to finding a map from $S$ \to $H$ such that the relations in $R$ are satisfied. For example $\Bbb Z/2\Bbb Z \times \Bbb Z/2 \Bbb Z = \langle a,b \mid a^2=b^2=1, ab=ba\rangle$, so for any group $H$, finding a homomorphism $\Bbb Z/2\Bbb Z \times \Bbb Z/2 \Bbb Z \to H$ is equivalent to finding two elements $x,y \in H$ that commute and have order at most $2$. This is much more methodic than considering all elements in $G$.
There are some special cases in which other useful facts apply:
If you consider maps from $G$ to an abelian group $A$, then any such much must factor through the abelianization $G^{ab}$ (this is the largest abelian quotient of $G$.), so you can reduce the problem to finding homomorphisms $G^{ab} \to A$. It can happen, for instance, that the abelianization of $G$ is trivial, then this means that there is no nontrivial homomorphism from $G$ to any abelian group. One example of this is $\operatorname{SL}_2(\Bbb F_5)$.
If you can decompose $H$ as a product $H=H_1 \times H_2$, then a homomorphism $G \to H$ is equivalent to a pair of homorphisms $G \to H_1, G \to H_2$