Let $f \in Hom(G, G')$, $f:G \rightarrow G'$
Moreover if $G, G'$ are sets such that : $(G, \star )$, $(G', \cdot )$are groups we have the following condition on $f$ :
$$f(x \star y) = f(x) \cdot f(y) $$ My question is then : why this particular condition on $f$ ? Why not for example : $f(x \star y) = x \cdot f(y)$ ?
And moreover why finding an homomorphism between two sets is important, does it helps proving something between two groups ?
On
$x\cdot f(y)$ is not defined, because $\cdot$ takes two elements of $G''$ and returns an element of $G''$. However, $x$ is not an element of $G''$, it is an element of $G$.
The condition, as it is, is a very natural condition as it basically says "this function doesn't mess up our operations".
Finding isomorphisms (i.e., bijective homomorphisms) between groups is very important, because it shows us that the two groups are basically the same (i.e. any algebraic property that holds for one group also holds for another).
On
What does $x\cdot f(y)$ even mean, since $x\in G$, and $y\in G'$?
Besides, homomorphisms are not defined between sets. They are defined between groups.
On
The other answerers have hinted you to your possible misconception. I want to tell you a bit about structure-preservation in general, because this is what an isomorphism does: it tells us that two things (mathematical objects) are essentially the same, just presented to us in two different ways. A homomorphism is a bit weaker, but I will come to it later.
I think graph theory provides us with a very accessible understanding of a structure to preserve. Look at these two networks (graphs):
Graphs consist of nodes and edges between them. The only important thing for graphs is what is connected to what. The actual representation of the graph does not matter at all.
The left example is a rectangle, or $4$-cycle. The right one is also a rectangle but twisted. In the sense of connectedness, these both graphs are structurally the same. But how to say this mathematically? We begin by introducing a notation for connectedness (adjacency): we write $v_1\sim v_2$ for the fact that the nodes $v_1$ and $v_2$ are connected by an edge. On the other hand we have $v_1\not\sim v_3$.
We now want to say that these both graphs are the same. We can show this by giving a way how to transform the left graph into the right one. This can be done by stating which vertex on the left should be mapped to which vertex on the right. We do this via a map $f$. For example we could say $f(v_i)=w_i$. However, this specific transformation is not structure preserving in the following sense: $v_1\sim v_4$ but $w_1\not\sim w_4$. We can state this by
$$v_1\sim v_4 \quad\text{but}\quad f(v_1)\not\sim f(v_4).$$
Just mapping each vertex with the index $i$ on the left to its corresponding vertex with index $i$ on the right will not give us a transformation that demonstrates the equality of the graphs. You see from this failure, that we need to claim the following for the map $f$:
$$v_i\sim v_j\quad\Rightarrow\quad f(v_i)\sim f(v_j).$$
This is the point where you should see the similarity to your group theory homomorphism. And indeed, such maps are called graph homomorphisms. Homomorphism are a very general concept. If two elements of an object have a specific relation to each other, then this relation should be kept when transforming the whole object onto another structurally equivalent object.
An example for a homomorphism would be
$$f(v_1)=w_1,\quad f(v_2)=w_2,\quad f(v_3)=w_4,\quad f(v_4)=v_3.$$
Note that we have not claimed $v_i\not\sim v_j\Rightarrow f(v_1i)\not\sim f(v_j)$ yet. This is indeed not necessary for homomorphisms. For example, the following is a valid structure preserving map for the left graph into the right one:
Many different nodes can be mapped to the same node, as long as they are not connected. Why? because a node is (in this kind of graphs) never connected to itself. So we obey the homomorphism rule. So what we actually found here is that the right "subgraph" consisting of the vertex $w_1$ and $w_3$ is somehow structurally similar to the complete left graph.
This is of course not the kind of identity we want to show with that both graphs are identical. For this we need isomorphisms. An isomorphism is a bijective map $f$, so that both $f$ and $f^{-1}$ are homomorphisms. They are structure preserving in both directions.
But back to group theory. For groups this is very similar. Instead of two elements (nodes) like for graphs, for groups we relate three elements via the expression $g_1\circ g_2 =h$. To preserve this structure via a map $f$ and carry it over to another group we have to claim $f(g_1)* f(g_2)=f(h)$. Now just substitute our former meaning of $h$ into the last equation:
$$f(g_1)* f(g_2)=f(g_1\circ g_2).$$
And this is a group homomorphism. Something that tells you that its image $f(G)$ is somehow similar to the former group $G$. As above, to really talk about two identical groups $G_1$ and $G_2$, you will have to consider bijections $f:G_1\to G_2$ so that both $f$ and $f^{-1}$ are homomorphisms $-$ so called group isomorphisms.
You probably confuse a homomorphism with something that you are familiar: A function $f:ℝ→ℝ$.
A homomorphism is more than that, because it does not only map numbers / elements of sets, but it connects two groups. And as you already stated in your Question: A group is a set with an operation $(G,\star)$. Therefore a mapping between two groups should be more than a mapping between two sets, right?
So the 'natural' thing to do, is to connect these two groups $(G,\star)$ and $(G',\cdot)$ in such a way, that it does not mess with the operation. Why? Because it would be a useless/random mapping. Keep in mind that mathematicians love structure - especially in algebra.
So what is 'not messing up everything'? It is the preservation of the only thing interesting that groups have, that sets don't have - the operation. So the mapping would read: $$ Φ:(G,\star)→(G',\cdot), \text{ such that } Φ(x\star y) = Φ(x)\cdot Φ(y)$$
In words this mapping tells you, that it does not matter if you first multiply two elements of $G$ and than map to $G'$ or if you multiply the mapped elements (in $G'$).
This mapping, that does not mess up, gives us, as it connects two groups, more structure. And, as I have mentioned above, mathematicians love structure - so they give that kind of mapping the name homomorphism.
If you like to draw diagrams, you can look at the following: $$ \begin{array}{ccc} G×G & \overset{\star}{→} & G \\ | & & | \\ Φ & & Φ \\ \downarrow & & \downarrow \\ G'xG' & \overset{\cdot}{→} & G' \\ \end{array}$$
Remark: You asked why not $Φ(x\star y) = x\cdotΦ(y)$. Remember how a group operation '$\cdot$' is defined: It is a mapping $\cdot:G'×G'→G'; (x,y)↦x\cdot y$. Hence, the expression $x\cdotΦ(y)$ is not well-defined if $G\not\subset G'$.
Now why are homomorphisms useful? We just defined them as 'not to mess everything up', or with other words 'preserve operations'.
Example: Homomorphisms preserve the identity element.
Let $e_G$ be the identity element of $G$, and $e_{G'}$ be the identity of $G'$. Then it is: \begin{align*} Φ(e_G) &= Φ(e_G\star e_G) \\ &=Φ(e_G)\cdot Φ(e_G) \end{align*} So we have: \begin{align*} e_{G'}\starΦ(e_G) =¹\ Φ(e_G) =\ Φ(e_G)\cdot Φ(e_G) \\ \end{align*} ¹ Since $e_{G'}$ is the identity element of $G'$.
And by cancellation law (of groups) we get: $$ e_{G'} = Φ(e_G). $$
In a nutshell you can say, that if group $G$ has a cool feature, and you can find / construct a homomorphism from $G$ to $G'$, then $G'$ has that cool feature as well. Or the other way round, if you want to proof a feature of group $G'$ and you have a group $G$ that has this feature - try to construct a homomorphism.