Although it is certainly important to study frequently seen group operations like permutations, function composition, word operations, and so on, I find it fascinating to see group structure applied in strange and complicated ways to other disciplines.
Lately MSE has got me looking at the group of functions $f:\mathbb{N}\rightarrow \mathbb{C}$ with $f(1)\not= 0$ under the Dirichlet product $\star$ given by $$(f\star g)(n)=\sum_{ab=n}f(a)g(b).$$ I think this is a great example of an unusual group operation. I don't usually get into this type of arcane number theoretic stuff and never would have thought to look at this type of structure myself.
What are some other examples of obscure group operations I may not have heard of?
The Butcher group is an infinite-dimensional Lie group arising in numerical analysis and ODEs. It is related to the Connes-Kreimer Hopf algebra, which appears in Connes and Kreimer's approach to understanding renormalization in quantum field theory.