I worked this exercise and found 86. My reasoning was as follows:
The rotations in $D_5$ are all of order 5 and thus can only go to the identity in $S_3$. The reflections are of order 2, so they can go to the transpositions $(1 \ 2), (1\ 3)$ and $(2\ 3)$ or the identity. So depending on how many reflections we send to the transpositions, let's say $n$, we get the following.
\begin{eqnarray} n=0 &\rightarrow& 1 \\ n=1 &\rightarrow& 5 \\ n=2 &\rightarrow& 5*4 \\ n=3 &\rightarrow& 5*4*3 \end{eqnarray}
So the total would be 86. Can someone confirm or explain why this would be wrong?
I didn't consider that you could send 2 reflections to the same transposition, so there are 4 choices for each reflection (id or one of the transpositions). So there would be $4^5 = 1024$ homomorphisms.
Recall that we can write
$$ D_5=\{ e, r, \dots, r^4, s, rs, \cdots, r^4s \} $$
where $r$ ($r^5=e$) is the rotation and $s$ ($s^2=e$) is a reflection. Let now
$$ \phi: D_5 \rightarrow S_3 $$
be a homomorphism of groups. We have
$$ \phi(r^n s^m)= \phi(r)^n \phi(s)^m. $$
Thus, $\phi$ is uniquely defined by $\phi(r)$ and $\phi(s)$. As we have
$$ id=\phi(e)=\phi(r^5)=\phi(r)^5 $$
and every element in $S_3\setminus \{id\}$ has order $2$ or $3$, we get that
$$ \phi(r)=id. $$
Hence, $\phi$ is uniquely determined by $\phi(s)$. Again we have
$$ id= \phi(e)= \phi(s^2)=\phi(s)^2. $$
Hence, $\phi(s)\in \{ id, (12), (13), (23)\}$ and therefore, there are exactly 4 group homomorphisms
$$ \phi : D_5 \rightarrow S_3. $$
Added: Actually one would need to check that all those choices really define a group homomorphism. The first option would be to check it by hand. Another way would be to use more abstract theory. For this one notes that we have the following representation of $D_5$
$$ D_5 \cong \langle r,s \vert srsr=e \rangle.$$
Therefore, $\phi: D_5 \rightarrow G$ ($G$ any group) defined by $\phi(r)=g_1, \phi(s)=g_2$ ($g_1, g_2\in G$) is a group homomorphism iff
$$ g_1 g_2 g_1 g_2 = e_G. $$
This holds for our cases, hence, there are really 4 group homomorphisms.