An action of a group $G$ on a set $X$ is a mapping $*:G\times X\rightarrow X$ such that
- $e*x = x$ for all $x \in X$
- $(ab)(x) = a(bx)$ for all $x \in X$ and all $a,b \in G$.
If one had a group and set that satisfied 1, but rather than satisfying 2 the following was satisfied instead:
$(ab)(x) = b(ax)$ for all $x \in X$ and all $a,b \in G$.
Consider an operation $*:SL_2(\mathbb{Z})\times \mathbb{F}$ where $\mathbb{F}$ is the set of all quadratic forms whose matrices have integer entries. That is, $\mathbb{F}$ is the set of all $Q:\mathbb{R}^2\rightarrow\mathbb{R}$ such that $Q({\bf v}) = {\bf v}^T\big[\begin{smallmatrix}a&b\\b&c\end{smallmatrix}\big]{\bf v}$, where $a, b, c$ are all integers.
Now let $Q$ have matrix $A$ and $M*Q({\bf v}) = Q(M{\bf v}) = (M{\bf v})^TA(Mv) = {\bf v}^T(M^TAM){\bf v} = Q'$. Then, $N*Q'({\bf v}) = (N{\bf v})^T(M^TAM)(N{\bf v}) = {\bf v}^TN^T(M^TAM)N{\bf v} = {\bf v}^T(MN)^TA(MN{\bf v}) = (MN{\bf v})^TA(MN{\bf v}) = Q((MN){\bf v}) = (MN)*Q({\bf v})$
And so $N*Q'({\bf v}) = N*(M*Q({\bf v})) = (MN)*Q({\bf v})$
Would that be of interest? Is there any documentation on such a structure?
You've just described a right group action! In a right group action, members of a group act on the right of elements in $X$ (as the name suggests). A right group action has the following associativity rule: $$(xa)b = x(ab).$$ Notationally, it is different, but it's functionally the same as what you have. The important part is that the action of $ab$ is equivalent to the action of $a$, followed by the action of $b$.
The theory of right group actions, though, is equivalent to that of left group actions. If we have a right group action, we can transform it into a left group action by taking $$ ax = xa^{-1}, $$ which obeys the associativity rule because $(ab)^{-1} = b^{-1}a^{-1}$. Thus, left and right group actions are, for the most part, equivalent.
Edited for the updated question: I might be missing something, but it looks like you could just use the notation of a right group action to describe this, so that $$ Q(\mathbf{v}) \ast M = Q(M\mathbf{v}). $$