I am learning Milne's Abelian Varieties. It gives the definition of coherent $G$-sheaf as following:
Let $V$ be a variety over $k$, $k$ is a field. Let $G$ be a finite group acting on $V$ (on the right) by regular map. A coherent $G-$ sheaf on $V$ is a coherent sheaf $\mathscr{M}$ of $\mathscr{O}_{V}$-module together with an action $G$ on $\mathscr{M}$ compatible with its action on $V$.
I do not really understand what is "an action $G$ on $\mathscr{M}$ compatible with its action on $V$ ". Now suppose $V=\mathrm{Spec}(A), \mathscr{M}=\widetilde{M}$, $A$ is a finitely generated $k$-algebra, M is finitely generated $A$-module. Then $G$ acts $A$ and $M$ on the left. I think there are two possible way to give the definition.
The first way is $g(am)=g(a)g(m)$ and the second way is $g(am)=ag(m)$.
Could you tell me which definition should choose? Also if some orbits of $G$ are not contained in open affine subvariety of $V$, we could not assume $V$ is affine. How should I understand the $G$- sheaf in this case?
Thank you very much for any help.
I would say that if $G$ acts on $\operatorname{Spec} (A)$, then $G$ has a right action by pullbacks on $A$. Then, what Milne means is that the compatible action is $g(am) = (g^*)^{-1}(a) g(m)$.$\DeclareMathOperator{\Spec}{Spec}$
In the non-affine case, a compatible action amounts to: for every $g\in G$, we give an isomorphism: $$ \overline g: \mathscr M\overset{\sim}\longrightarrow g^*\mathscr M. $$ (Caution: some people use $\overline g:g^*\mathscr M\to \mathscr M$ instead, and I'm not sure what convention Milne follows). There is an extra condition, which corresponds to the property $g(hx)=(gh)x$ of an action. For every $g,h\in G$, we need to have $$ g^*(\overline h)\circ \overline g=\overline {hg}:\mathscr M\longrightarrow g^*\mathscr M \longrightarrow g^*(h^*\mathscr M) = (hg)^*\mathscr M. $$ I personally find this definition pretty unintuitive though. Here's some facts that make it a bit easier to digest, at least for me:
The problem with the side of the action is that for a map of schemes $g:\operatorname{Spec}(B)\to \operatorname{Spec}(A)$, the corresponding map of rings goes in the opposite direction $g^{\#}:A\to B$, and it's given by $f\mapsto f\circ g$. I guess when you write something like $g(a)$ it could mean several things.
For your second question, the answer is similar: pullbacks compose in the opposite direction, so $(g\circ h)^*\cong h^*\circ g^*$. $ (gh)^*\mathscr M$ can't equal $g^*h^*\mathscr M$. Imagine these were different spaces and we had: $$ \newcommand{\onarrow}[1]{\overset{#1}\longrightarrow} X\onarrow{h} Y \onarrow{g} Z. $$ Then the pullbacks map this way: $$ \DeclareMathOperator{\Coh}{\mathbf{Coh}} \Coh(Z) \onarrow{g^*} \Coh (Y)\onarrow{h^*} \Coh (X). $$ So $g^*\circ h^*$ doesn't even make sense: the target of $h^*$ is not the domain of $g^*$. Only $h^*\circ g^*$ makes sense.
$\newcommand{\M}{\mathscr M} \newcommand{\eps}{\varepsilon} \newcommand{\ov}[1]{\overline{#1}}$ Now for the affine case. In this case, the pullback $g^*\mathscr M$ is $A\otimes_{g} \M$, i.e. the tensor with this relation: $$ g^*a\otimes m = 1\otimes a m. $$ Or, if we write $g^*m = 1\otimes m$, we can write it as: $$ g^*a\cdot g^*m = g^*(am). $$ There is a bijection $g^*:\M\to g^*M$, given by $m\mapsto g^*m = 1\otimes m$. Now, this bijection is not $A$-linear. Instead, $g^*\circ a = g^*a\circ g^*$: $$ g^*(am)=1\otimes am = g^*a\otimes m = g^*a\cdot g^*m.$$ We can replace the maps $\ov g:\M\to g^*\M$ by their compositions with the maps $g^*\M\to \M$. Let's call them $\widehat g = (g^*)^{-1}\circ\ov g$.
The maps $\ov g$ are defined to be $A$ linear. Therefore, $$ \widehat g(g^*a\cdot m) = (g^*)^{-1}(g^*a\cdot \ov g(m)) = a(g^*)^{-1}( \ov g(m)) a\widehat{g}m. $$ Conversely, if the maps $\widehat g$ satisfy $\widehat g\circ g^*a = a\circ \widehat g$, this ensures that $\ov g := g^*\circ \widehat g$ will be $A$-linear.
Now, for the associative property: $$ \widehat h\circ \widehat g=(h^*)^{-1}\circ \ov h \circ (g^*)^{-1} \circ \ov g $$ Note that for a map of sheaves $\phi$, $g^*\phi = g^* \circ \phi \circ (g^*)^{-1}$. So $$ (h^*)^{-1}\circ \ov h \circ (g^*)^{-1} \circ \ov g = (h^*)^{-1}\circ (g^*)^{-1} \circ g^*\ov h\circ \ov g = ((hg)^*)^{-1} \circ \ov{hg} = \widehat {hg}. $$ So the relation $g^*\ov h\circ \ov g$ that we imposed above implies $\widehat{hg} = \widehat h\widehat g$. The same computation shows the reverse implication.
So with this discussion in mind, a $G$-sheaf on $\Spec A$ is equivalent to an $A$-module $M$ together with a left $G$-action such that $$ g(g^*a\cdot m) = a\cdot g(m). $$