The following is just a basic definition in Invariant Theory, which I copied from wikipedia
"Let $G$ be a group, and ${\displaystyle V}$ a finite-dimensional vector space over a field ${\displaystyle k}$ (which in classical invariant theory was usually assumed to be the complex numbers). A representation of ${\displaystyle G}$ in ${\displaystyle V}$ is a group homomorphism ${\displaystyle \pi :G\to GL(V)}$, which induces a group action of ${\displaystyle G}$ on ${\displaystyle V}$. If ${\displaystyle k[V]}$ is the space of polynomial functions on ${\displaystyle V}$, then the group action of ${\displaystyle G}$ on ${\displaystyle V}$ produces an action on ${\displaystyle k[V]}$ by the following formula:
${\displaystyle (g\cdot f)(x):=f(g^{-1}(x))\qquad \forall x\in V,g\in G,f\in k[V].} $"
I did not quite understand the group action at the end. I mean, we need to prove $((gh)\cdot f)(x)=g\cdot (h\cdot f)(x)\qquad \forall x\in V,g,h\in G,f\in k[V]$
I have two ways to understand the right hand side:
1) $g\cdot (h\cdot f)(x)=g\cdot f(h^{-1}(x))=f(g^{-1}h^{-1}x)$
2) $g\cdot (h\cdot f)(x)= (h\cdot f)(g^{-1}x)=f(h^{-1}g^{-1}x)$
Of course, to make this a group action, the second way is correct. However, I want to ask how can one just look at $g\cdot (h\cdot f)(x)$ and tell which way is correct? In fact, I think the first way is more rational as we need to compute $h\cdot f$ first.
Thank you in advance
$ g \cdot (h \cdot f) (x) $ is by definition the evaluation of $ g \cdot (h \cdot f) $ on $ x $ .
This function is of the form $ g \cdot m $ for $ m = h \cdot f $, thus by definition it evaluation on $ x $ is $ m(g^{-1}x) $ .
This is the evaluation of $ h \cdot f $ on $ g^{-1} x $ which is by definition $ f(h^{-1} (g^{-1}x)) =f (h^{-1}g^{-1}x) $ .
So there is only one way to evaluate this, the other one is a common mistake.