I've always seen this definition of a group $G$ acting on a set $\Omega$, making this latter a $G$-set:
Given a group $G$ and a set $\Omega$ we say a group acts on the set $\Omega$ when there's a function $\Omega\times G \mapsto \Omega$ such that, denoting with $\alpha^g$ the image of $(\alpha,g)$ it holds:$$\alpha^{gh}=(\alpha^g)^h, \alpha\in\Omega, g, h \in G \\ \alpha^e=\alpha, e=G_{Id}$$
In particular this definition applies also if $\Omega$ is yet another group $H$, for instance.
So far, so good. Recently, studying the group extension problem and the short exact sequence, I faced the cohomology and, trying to get a grasp out of it (even if a bit too advanced for my current group training), I came to this definition:
Given two groups $G$, $H$ and a homomorphism $\phi:G \mapsto {\rm Aut}(H)$, we say G acts on H through $h^\sigma=h^{\phi(\sigma)}, h \in H, \sigma \in G$
I was wondering if these two definitions are equivalent and the second should be preferred to the former, when the target set is a group as well, since more coincise and using ${\rm Aut}(H)$ without the need to specify the other conditions in the former definition.
Thank in advance.
The definitions are not equivalent. The difference stays in the fact that on a $G$-set $X$ the definition of action is equivalent to have a homomorphism $$G\to S(X)$$ where $S(X)$ is the group of self-bijections of $X$. If $X$ has a group structure the second definition is equivalent to have a homomorphism $$G\to Aut(X)$$ that is a stricter condition.
For instance consider $X=\Bbb Z/4\Bbb Z$ and $G=\Bbb Z$. Here the action induced by $1\cdot [1]=[2]$ is an action according to the first definition and not according to the second: an automorphism can send $[1]$ only to an other generator of $X$.
However you will often find the second definition applied to groups: more generally, a group $G$ action on some object $V$ of some category will be often asked to be an homomorphism $G\to Aut(V).$ This thus applies also when $V$ is a vector space, or a ring, or an algebra!