Suppose that you have some group $G$ acting on a set $S$ under the action $\cdot: G \curvearrowright S$, and then you have a set of all functions $T = \{ f: S \to \mathbb{C} \}$. I want to define an action $*: G \curvearrowright T$.
Let $\omega_g: s \in S \mapsto g \cdot s$, then I want to define $g * f(x) = f(g \cdot x)$. This seems to be associative:
$a * ( b * f(x)) = a *(f(b \cdot x)) = f(a\cdot (b \cdot x)) = f((ab) \cdot x) = ab * f(x)$
However, when I try to write this action out in terms of $\omega_g$ it gets problematic (I'll denote the action here by $\times$ for clarity)
$a \times ( b \times f(x)) = a \times ( f \circ \omega_b) = (f \circ \omega_b) \circ \omega_a = f \circ \omega_{ba} = ba \times f(x)$
These two actions are obviously not the same, but I can't seem to write my first action in the $\omega_g$ notation.
I am wondering whether the first action is really an action, or whether the notation is hiding something which makes it not an action. Moreover, how can it be written in the $\omega_g$ notation? I can fix the omega notation by mapping to $\omega_{g^{-1}}$, but paradoxically it doesn't seem to be associative if I use my first notation.
Your "associative property" is incorrect. In fact, if $(g\ast f)(x):=f(g\cdot x)$ then
$$ (a\ast(b\ast f))(x)=(b\ast f)(a\cdot x)=f(b\cdot (a\cdot x)),$$
which is "backwards." Indeed if you define $h(x)=f(b\cdot x)=(b\ast f)(x)$ then
$$ (a\ast (b\ast f))(x)=(a\ast h)(x)=h(a\cdot x)=f(b\cdot (a\cdot x)). $$
The reason things look backwards is that $(g\ast f)(x):=f(g\cdot x)$ does not define a left action of $G$ on the function space $T$, it defines a right action. It would be better to write $(f\ast g)(x)=f(gx)$.
More often, sources will define a left action of $G$ on $T$ via $(g\ast f)(x):=f(g^{-1}\cdot x)$. This is called the contragredient action. This left-vs-right issue is recurring in some other places too...
Example 1. Consider transformations of graphs of functions. If we begin with a graph of $y=f(x)$ then to move it up or down respectively we graph $y=f(x)+c$ and $y=f(x)-c$. And yet, contrary to intuition, to move the graph to the right by $c$ units (where $c>0$), we get the graph
$$\{(x+c,f(x)):x\in\mathbb{R}\}=\{(x,f(x-c)):x\in\mathbb{R}\}. $$
That is, to move the graph to the right, we graph $y=f(x\color{Red}{\mathbf{-}}c)$ even though on the number line we know that $x\mapsto x+c$ moves right and $x\mapsto x-c$ moves left. This issue confounds many a student of college algebra.
Example 2. Consider the set $[n]=\{1,\cdots,n\}$. Then the function space $X^{[n]}$ may be identified with the set of $n$-tuples with entries in $X$. (Any function $f:[n]\to X$ can be thought of as the tuple $(f(1),\cdots,f(n))$ and vice-versa.) There is an action of the symmetric group of $S_n$ on $[n]$ using permutations, and hence on $n$-tuples. Naturally, we want $\sigma\in S_n$ to permute the coordinates of a tuple the same way it permutes $\{1,\cdots,n\}$. So for instance, we would want $\sigma=(123)$ to send shift the entries of $(x_1,x_2,x_3)$ rightwards, yielding $(x_3,x_1,x_2)$. The problem is that this is not $(x_{\sigma(1)},x_{\sigma(2)},x_{\sigma(3)})$, instead it's $(x_{\sigma^{-1}(1)},x_{\sigma^{-1}(2)},x_{\sigma^{-1}(3)})$.
This example also occurs with $S_n$ acting on the tensor power $V^{\otimes n}$. In my experience I've seen numerous authors of textbooks and lecture notes mistake left and right actions in this context.