I'm confunsed by the definition of direct image of sheaf which says
Let $f :X \rightarrow Y $ be a continuous map of topological spaces, and let $\mathcal{F}$ be sheaf on $X$. Then there is a sheaf on $Y$ given by $(f_{*} \mathcal{F})(V)=\mathcal{F}(f^{-1}(V))$, where $V$ is an open subset of $Y$.
But $(f^{-1}(V))=U$, and $U$ in an open subset of X. The problem is we will get a sheaf on $X$, not on $Y$. A sheaf on $Y$ should be defined on some open subset of Y and gives as some nice function(holomorphic,....) on that open subset, but here by that definition we get $\mathcal{F}(U)$ which is a collection of functions on $U$ and they have no relation with functions on $V$!
Can some one explain why this definition is true, or give an example to make it clear?
Just like you said, we need to assign open set $V$ of $Y$ some set or abelian group or whatever. Let do sets here. So what set are we assigning to $V$? We are assigning this set: $$\mathcal{F}(f^{-1}(V))$$ Is this a set? Yes it is. If I give you an open set $V$ of $Y$, can you tell me what the above set is? Yes because you know what $\mathcal{F}$ is. So, this indeed defines a sheaf on $Y$ (You need to check the sheaf definitions, but that's not what your question is about).
As an example, consider a topological space $X$, and a point $p \in X$. Consider $\{p\}$ as a space with only one point. Let $S$ be a set and let $\underline{S}$ be the constant sheaf on $\{p\}$. Let $f: \{p\} \rightarrow X$ be just the inclusion. Then we can do the direct image sheaf. For any open set $V$ in $X$, if $V$ contains $p$ that $f^{-1}(V) = \{p\}$, and if not $f^{-1}(V)$ is empty. So the direct image sheaf would be $$f_{*}\underline{S}(V) = S \text{ or } \{?\}$$ where $\{?\}$ is any singleton set. This is the skyscraper sheaf on $X$ at $p$.