My book says that a permutation is a bijective mapping from a group to itself.
So, let our group be $S$ and let our function be $Y(y)=xy$ where $x,y\in S$.
Now we know that since $S$ is a group, $xy \in S\quad \forall y\in S$.
Now testing injectivity:
Suppose we have $Y(a_1)=Y(a_2)$. This implies $xa_1 = xa_2$ and $a_1 = a_2$ after left multiplication by $x^{-1}$.
Now testing surjectivity:
Suppose we have $a\in S$. We can get this by $Y(x^{-1}a)=xx^{-1}a$ so the function is surjective.
But how is the group $S$ being permuted over? I understand permutations to be a select number of elements from a group put into a particular order.
In our case the function gives rise to many "infinite cycles" such as:
$$(e, x,x^2,x^3,x^4,...)$$
So the way I undertand this is that if $R$ is a subset of $S$ then $Y$ applied to $R$ would map each element in $R$ to the coset $xR$. Is that it?
This map is indeed a permutation, because in Group Theory (and generally for any set) the definition of a permutation is exactly the same as a bijection from a set to itself.
It sounds like you're using a definition of permutation that has more to do with combinatorics - choosing a certain subset of elements in a certain order. In group theory, a permutation is different; you are given a set of elements, and they are `shuffled' in some way.
To see what the permutation is doing to a finite group, try writing out its multiplication table. Then the row corresponding to left multiplication by $x$ shows exactly how every element is being shuffled;
$e\mapsto x, x^{-1}\mapsto e$ etc.
Try this with a few specific examples, like $\mathbb{Z}_3$.
You're also correct about mapping $R\mapsto xR$; the insight here is that, because $x$ has an inverse, $Y$ must be an invertible function, and so $xG=G$, for any group $G$ and any element $x$.
Note: I'm not sure what you're thinking about with the idea of an infinite cycle; it's true that you can let $R$ be the subgroup generated by $x$ i.e. $R=\{\ldots, x^{-2}, x^{-1}, e, x, x^2\ldots\}$. Then $Y$ permutes this set in a very tidy way, moving every element one step to the right. You'll find something very similar happens in the example of $\mathbb{Z}_3$, regardless of which element you choose.