We are asked to find the number of homomorphisms from $G$ to $S_5$. We consider the homomorphisms from $F_2$ (with generators $x$ and $y$) to $S_5$ and then we say that if image of $x$ and $y$ satisfy the relations, then this can be factored by a normal subgroup containing the relation.
The question is how to formally find the number of such homomorphisms $\phi: F_2 \to S_5$ satisfying $x^{-1}yx=y^6$ (I understand that the idea is to observe the order of $y$ and determine if $y$ is a $2, 3, 4, 5$ - cycle, but what to do with $x$) and how to act if (for example) the number of cases to consider for the images of $x, y$ is large?
Here are some thoughts which should surely get you started. Conjugation in the symmetric group preserves the cycle type, because it has the effect of permuting the underlying symbols. Thus, the relation $x^{-1}yx = y^6$ tells you that $y$ and $y^6$ have the same cycle type. Then we see which cycle types this is possible for. Since the cycle decomposition breaks down into disjoint cycles, this can only happen when each cycle in the decomposition has an order coprime to $6$, else raising it to the sixth power will result in some trivial cycles appearing, or else modify the cycle type, which we can't have. Since $6 = 2 \cdot 3$, this forces $y$ to be either a trivial cycle or a cycle of length $5$, as no other orders can appear. If it's the trivial cycle, then you can send $x$ anywhere. So one family of maps comes from $y \mapsto e, x \mapsto \text{anything}$ and this finds us $5!$ uninteresting maps.
Can you handle the other case?