A dihedral group seems to have 2 operations, rotation, and transposition.
These transformations seem so distinct, no composition of rotations may ever lead to a transposition and vice versa (right?).
so why can we use the same operation symbol (like $ \circ $) for these 2 seemingly different operations? are they connected in some deeper way?
No. The dihedral group has two types of elements (not operations), which are rotations and reflections (not transpositions). You are right about one thing, though: no composition of rotations may ever lead to a reflection. And... ? We also have two types of integers: odd and even. Furthermore, no sum of even integers may ever lead to an odd integer. Do you have any problem with this? It seems exactly the same situation.