By Cayley's theorem every abstract group is isomorphic to some permutation group. Since the permutation group viewpoint has the advantage of considering the actions of the group on different sets, and therefore, of finding structure not just in the underlying set of the group, but in the behaviour of its elements, why do we not always consider the permutation representations of a group?
In other words, is there an advantage to looking at "groups" proper rather than groups of permutations? Is the answer different for finite and infinite groups?
(Edit) In the comments, it was pointed out that the details introduced by considering a particular permutation/linear representation may be a hinderance when, for example, the focus is on combinatorial properties of groups given as their group presentation. What are other examples of situations in group theory when the abstract view of the group is preferred?
Extended comment:
in history (finite) permutation groups were discovered much before groups (in modern language: set of permutations of a set including the identity and stable under composition). Note that the notion of abstract group isomorphism was already known at that time.
The transition to abstract groups was discovered around 1880, and simultaneously infinite groups were discovered. (It was not yet obvious that "group" would not eventually mean what "monoid" actually means, and I've actually seen a paper from the 1910s (by Andreoli) writing "group" for "monoid".)
The abstract definition, for finite groups, has helped in considerable progress notably by Frobenius and Burnside, on representation theory, and subsequent ones.
Some traces of this history are visible now: for instance in the pre-abstract area one had to distinguish, in a group, the number of letters, and the number of "substitutions". This is why we call the "order" of a finite group rather than its cardinal.
(In contrast infinite groups were born already with the abstract point of view, partly due to the fact that the notion of infinite set was somewhat not well-admitted.)
Since then the teaching is traditionally exclusively focussed on groups in its abstract definition, but this makes it much more abstract and one could imagine teaching otherwise (I'm not aware of modern experiments in this direction.) Indeed mathematicians of that time had difficulty with this step of conceptualization (law on an abstract set, associativity etc), so it's not surprising students do as well.