Tradition in physics tends to match the concept of group with that of "symmetry". Are there good and/or profound reasons to do so?
If one sticks with the intuitive content of "symmetry", this word should be reserved to orthogonal and unitary groups. Ok, many abstact (Lie) groups have nice representations in unitary groups, but in general none of these representaiions is injective, so the group is not isomorphic to a subgroup a unitary group.
In those cases, isn't it an abuse of language to still refer to the members of such a Lie group as "symmetries"?
And even worse: many people do so not only with Lie groups, but with the general concept of group.
Symmetries are maps from an object to itself which preserve some structure. The identity map is always a symmetry, the composition of two symmetries is a symmetry, and symmetries are generally invertible (either because a non-invertible map would destroy the structure, or because we decree that symmetries need to be bijective).
Therefore, the collection of symmetries of any object naturally forms a group, with the operation of composition.
If you have four identical particles, for example, any permutation of those particles is a symmetry, since the state before the permutation is indistinguishable from the state after. So we say that $S_4$ is the group of symmetries of $4$ identical objects. There's no need to restrict ourselves to unitary and orthogonal groups. Not all symmetries are transformations of inner-product spaces.