My professor wrote at the beginning of speaking about group actions this:
In general, Aut$(X) \subset $ Sym$(X)$ acts on $X$. If $G \subset Aut(X)$ is a subgroup, we say that "G acts on $X$ by appropriate automorphism."
Then he gave us a first definition for Group action which is: If $G$ a group, $X$ a set. a group action by $G$ on $X$ is a function : $G \times X \rightarrow X$ defined by $$(g,x) \mapsto {}^gx$$ such that $$ {}^g({}^hx) = {}^{(gh)}x$$ for all $g, h \in G.$
Then he gave us a second definition which is: any group homomorphism $G \rightarrow Aut(X).$
Then he gave us examples as follows:
If $V$ is a vector space over $k$ of dim. $n < \infty.$
1- $GL_{n}(k) = GL(V)$ acts on $V$ by linear transformations. $SL_{n}(k)$ acts on $V$ by restriction.
My questions are:
1- I do not understand how $Aut(X)$ acts on $X.$ what is the implied operation in that case?
2- I do not understand how is the second definition is also a group action definition? what is the implied operation in that case?
3- How can I prove that the example given is really a group action?
Could anyone help me answer those questions please?
For your first question: Going by the first definition, an action of $\operatorname{Aut}(X)$ on $X$ should be a function $$\operatorname{Aut}(X)\times X\ \longrightarrow\ X:\ (f,x)\ \longmapsto\ {}^fx,$$ satisfying ${}^g({}^hx)={}^{(gh)}x$. So to every pair $(f,x)\in\operatorname{Aut}(X)\times X$, we associate a new element ${}^fx\in X$. What is the obvious choice of element to associate with the pair $(f,x)$? Can you show that this association satisfies ${}^g({}^hx)={}^{(gh)}x$?
For your second question: Given a group homomorphism $$\varphi:\ G\ \longrightarrow\ \operatorname{Aut}(X),$$ for every $g\in G$ its image $\varphi(g)\in\operatorname{Aut}(X)$ is an automorphism of $X$, so in particular a map from $X$ to $X$. For notational clarity define $\varphi_g:=\varphi(g)$ for all $g\in G$, which is an automorphism of $X$ for each $g\in G$. Then $$\psi:\ G\times X\ \longrightarrow\ X:\ (g,x)\ \longmapsto\ \varphi_g(x),$$ defines a group action of $G$ on $X$. You should verify this from the definitions of group homomorphism and group action.
Once you have verified and understood these two answers, I think you should be able to answer the third question by yourself; it would at least be a good exercise to try it (again).