It is given that $G:X$ and $G:Y$. Does this $[g\bullet f](x) := g\bullet f(g\bullet x)$ formula define group action $G:(Y^{X})$ I guess it doesn't, but I can't prove it as for now. And there must be some examples that show it.
On the contrary, this formula $\forall g\epsilon G;[g\bullet f](x):=g*f(g^{-1}*x),f\epsilon Y^{X}, x\epsilon X$ is correct, but I fail to comprehend it completely. I'm also curious how it (the correct one) was derived from the basic formula of group action on set. P.S. $G : X$ notation stands for group action on a set.
So $G$ acts on sets $X$ and $Y$ (on the left). On the set $Y^X$ of all maps $X \to Y$ we try to define an action of $G$. Of course $(gf)(x):=g f(gx)$ doesn't work since you can immediately check that this doesn't satisfy $(gh)f = g(hf)$. But when you define $(gf)(x):=g f(g^{-1} x)$, then we have $1f=f$ (clear) and $(gh)f=g(hf)$ , since for all $x \in X$ we have $$g(h(f))(x)=g h(f)(g^{-1}x) = g (h f(h^{-1} g^{-1} x))=gh f((gh)^{-1} x) =((gh)f)(x)$$ This action might seem strange at first sight, because the inverse is involved. But it has the nice property that it is, in fact, an action, and notice that the fixed points (i.e. those functions $f$ with $gf=f$ for all $g \in G$) are exactly the $G$-equivariant maps $X \to Y$. In symbols, we have $\hom(X,Y)^G = \hom_G(X,Y)$. This is quite useful.
Also notice the following more general construction: Let $G$ act on $X$ on the right and let $H$ act on $Y$ on the left. Then $Y^X$ has an action from $G \times H$ on the left, defined by $((g,h)f)(x):=h f(xg)$. By taking inverses, any right action can be turned into a left action, and vice versa. But this inverse-free construction works in great generality (closed symmetric monoidal categories where $G,H$ are replaced by arbitrary monoid objects), a special case being modules over rings. Notice that the right action of $G$ on $X$ becomes a left action of $G$ on $Y^X$ because $X \mapsto Y^X$ is a contravariant functor.