I am looking for a definition of a ring of formal power series, just like $R[[X]]$, where $R$ is a commutative ring, but with another group $G$ built in. Let $G$ be a group. In other words it is not clear to me what the transition
$R[G]\rightsquigarrow R[[G]]$
from group rings to "something" would be, which should be analogous to the transition from the ring of polynomials to the ring of formal power series
$R[X]\rightsquigarrow R[[X]]$.
In fact it would be enough for me to know such a definition for $R=\mathbb{Z}$.
Let $R$ be a commutative ring and let $G$ be a group (or more generally a monoid). There is a thing that deserves to be called a completed group algebra $R[[G]]$, although it may not be quite what you expected. As Espen says, in general given a ring $S$ and a two-sided ideal $I$ of that ring there is an $I$-adic completion given by the cofiltered limit of the quotients $S/I^n$. When $S = R[x_1, \dots x_n]$ is a polynomial ring we can take $I = (x_1, \dots x_n)$ and the resulting completion is a formal power series ring.
When $S = R[G]$ is a group algebra there is a different distinguished ideal to care about, namely the augmentation ideal, generated by the elements $g - 1$ for all $g \in G$. The idea intuitively is to think of all of the elements of $G$ as being close to the identity, and hence to think of all of the elements $g - 1$ as being small. Among other things, if $R$ contains $\mathbb{Q}$ then this allows us to make sense of the formal logarithm
$$\log g = \sum_{n \ge 1} \frac{(g - 1)^n}{n}$$
of every $g \in G$.
This is one step in the construction of the Malcev completion. Note that if we set $G = \mathbb{Z}_{\ge 0}$, so that $R[G] \cong R[x]$, then $R[[G]]$ is not $R[[x]]$; rather, it is $R[[x - 1]]$.