Let $R$ be a ring and $G$ be a group of ring automorphisms of $R$. Let consider the additive group $P=\bigoplus_{\sigma\in G}R$ with multiplication defined for $x,y\in P$ by $$(xy)_\omega=\sum_{\sigma\tau=\omega}x_\sigma \sigma^{-1}(y_\tau)$$ for every $\omega\in G$, where the sum runs over $\sigma,\tau\in G$ such that $\sigma\tau=\omega$.
This algebraic structure has a name? It is studied anywhere?
On
If we use the usual conventions, the multiplication defined by the OP is not associative when $G$ is not Abelian. If we drop the inverse in the definition of multiplication (i.e., use $x_{\sigma}\sigma(y_{\tau})$), then this construction is the skew group ring (denoted many ways, $R*G$ being one notation). The latter is the ring described in @guidoar's answer. I am unaware of a term for the OP's object with the non-associative multiplication.
If we introduce a placeholder $\overline g$ for every $g\in G$, it is common to denote a typical element of $P$ as $\sum_{g\in G}r_g\overline{g}$. (In @guidoar's notation, $r_g\overline{g}$ is $r_g\rtimes g$.) In this form, the OP's multiplication is defined by the rule $r\overline{g}\cdot s\overline{h}=rg^{-1}(s)\overline{gh}$. To see that this need not be associative, suppose $gh\ne hg$. This means there is an $r\in R$ with $g(h(r))\ne h(g(r))$. Let $e$ be the identity of $G$ and set $x=1\overline{g^{-1}}$, $y=1\overline{h^{-1}}$, $z=r\overline{e}$. Then $(xy)z=(g^{-1}h^{-1})^{-1}(r)\overline{g^{-1}h^{-1}}=(hg)(r)\overline{g^{-1}h^{-1}}$, while $x(yz)=g(h(r))\overline{g^{-1}h^{-1}}$. Since $h(g(r))\ne g(h(r))$, we see $(xy)z\ne x(yz)$.
One way to fix the associativity is to omit the inverse, that is, define $r\overline{g}\cdot s\overline{h}=rg(s)\overline{gh}$. This yields the skew group ring $R*G=R\rtimes G$. If we are wedded to the inverse, here a couple of ways to fix this. One is to use non-standard conventions; for example, take $\sigma\circ\tau$ to mean do $\sigma$ first, then $\tau$. In this case, it's natural to write arguments on the left, so $(r)(gh)=((r)g)h$. With this convention, we have $(xy)z=x(yz)$ in the above example. Another way to retain the inverse is to change the multiplication as follows. First, write elements of $P$ as $\sum_{g\in G}\overline{g}r_g$ and then define multiplication via $\overline{g}r\cdot \overline{h}s=\overline{gh} h^{-1}(r)s$. We can also think of this as $\overline{h}s=h(s)\overline{h}$, and it is another version of the skew group ring. In the OP's notation, the product $xy$ becomes $(xy)_{\omega}=\sum_{\sigma\tau=\omega} \tau^{-1}(x_{\sigma})y_{\tau}$.
If $R$ is a $G$-ring, then its crossed product $R \rtimes G$ is the $R$-module $R \otimes_{\Bbb Z} \Bbb Z G$ together with the multiplication determined by
$$(a \rtimes g) \cdot (b \rtimes h) = a \cdot g(b) \rtimes gh.$$
Here $a \rtimes g$ is notation for $a \otimes g$. Notice that $R \rtimes G$ is $G$-graded, $(R \rtimes G)_g = R \rtimes g = \mathsf{span}_{\Bbb Z}\{r \rtimes g : r \in R\}$.
Just to emphasize the similarities with the question asked, I will write the homogeneous component of degree $g \in G$ of $x \in R \rtimes G$ as $x_g \rtimes g$.
If $x = \sum_g x_g \rtimes g, y = \sum_h y_h \rtimes h$ then
$$ xy = \sum_{g,h} (x_g \rtimes g)(y_h \rtimes h) = \sum_{g,h}x_g g(y_h) \rtimes gh $$
Thus, for a given $\omega \in G$ the degree $\omega$ component of $xy$ is
$$ \sum_{\omega = gh} x_g g(y_h) \rtimes \omega. $$
This is not quite your multiplication, but it is similar (the difference being that $g$ should be $g^{-1}$). Perhaps there's a way to add an $(-)^{op}$ somewhere to make things work.