I know this question may seem a bit silly and trivial, but I haven't found any precise definitions for what a "symmetry" is in my textbook for group theory or online and I feel like I don't know it rigorously enough. Specifically, the root of my question is:
Why doesn't the dihedral group of order 2n have n! elements?
For example, the symmetries of a square (that is the dihedral group of order $2n = 8$, 4 reflections and rotations) has order $8$, but why isn't, for example $(12)$ a valid element in it?
I understand that there are axes of rotations and symmetries, but is that it? Is there a more precise definition of what a "symmetry action" must constitute?
You've asked several questions. I'll start with the easy one.
If you imagine a square with vertices labeled $1,2,3,4$ (in cyclic order) there are just eight permutations of that set that respect the geometry of the square. $(1,2)$ isn't one of them - you can't interchange two adjacent vertices and leave the other two fixed. (You can do that with diagonally opposite vertices).
The second question asks for
The precise definition depends on the structure whose symmetries you are trying to describe. Roughly speaking, its a bijective transformation that preserves all the properties that matter at the moment. For a square, those properties are the angles and the lengths of the sides. Then you prove geometric theorems that say the only operations with that property are the eight rotations and reflections.
The definition of a group and the theorems about groups have their roots in the struggle to make the idea of symmetry precise. Sometimes that history is lost when the study of abstract algebra begins with the definition.