the symmetry group of a geometric object is the group of all transformations under which the object is invariant, endowed with the group operation of composition
So why does the symmetry group of a square include only rotations and reflections? If you applied a transformation in the square that for example switched the position of two arbitrary points, it would not be a reflection nor a rotation. So why isn't that transformation included in it's symmetry group?
On
Because we only allow the "isometry", a transformation that preserves the distance of any two points.
For example, when you have a square piece of paper, the transformations you can do (such that the paper still looks like a square as before) are only the rotations and reflections.
On
If you applied a transformation in the square that for example switched the position of two arbitrary points, it would not be a reflection nor a rotation. So why isn't that transformation included in it's symmetry group?
Then it would not be the same object. For example, let us say I applied the following transformation of a square:
I have swapped two arbitrary vertices: $A$ and $B$. In the shape on the left, vertex $A$ could only be next to vertex $B$ or vertex $D$. Also, $B$ can only be next to $A$ or $C$. $A$ could never be next to vertex $C$; $B$ can never be next to $D$. If $A$ were next to $C$ or $B$ were next to $D$, then we would be talking about a completely different shape.
Indeed, on the picture on the right, by swapping just $B$ and $A$, I have violated those properties. I have produced a completely new shape. Hence, this transformation does not produce an invariate symmetry, in the sense that while it looks the same, it is not the same object.
If you include such permutations of swapping arbitrary vertices, then you would generate the entirety of $S_4$. In the symmetry group of a square, we only consider permutations of the symmetries of a single object, without changing that object's fundamental characteristics.
On
the symmetry group of a geometric object is the group of all transformations under which the object is invariant, endowed with the group operation of composition
[emphasis added]
If you look at the entry for "invariant, you'll see this:
In mathematics, an invariant is a property, held by a class of mathematical objects, which remains unchanged when transformations of a certain type are applied to the objects. The particular class of objects and type of transformations are usually indicated by the context in which the term is used. For example, the area of a triangle is an invariant with respect to isometries of the Euclidean plane. The phrases "invariant under" and "invariant to" a transformation are both used. More generally, an invariant with respect to an equivalence relation is a property that is constant on each equivalence class.
So the meaning of "invariant" is "indicated by the context". It's mentioned that for a triangle, the meaning is taken to be isometries. There is further confirmation that of this later in the symmetry article:
The group of isometries of space induces a group action on objects in it, and the symmetry group Sym(X) consists of those isometries which map X to itself (as well as mapping any further pattern to itself). We say X is invariant under such a mapping, and the mapping is a symmetry of X.
So what does "isometry" mean? It comes from "iso", which means "same", and "metric", which means "measure" or "distance".
In mathematics, an isometry (or congruence, or congruent transformation) is a distance-preserving transformation between metric spaces, usually assumed to be bijective.
So if you switch two point in a square, does that preserve distances? No. If you switch points A and B, then a point that was close to point A will not be close anymore. In two dimensional space, once you decide where three non-colinear points go, each other point has only one place it can go for distances to be preserved. That is, for any points A, B, and C, once you decide where f(A), f(B), and f(C) are, there's only one position for f(D) such that the distance from f(A) to f(D) is the same as the distance from A to D, the distance from f(B) to f(D) is the same as the distance from B to D, and the distance from f(C) to f(D) is the same as the distance from C to D. This can be shown by drawing a circle centered at f(A) with radius AD, another centered at f(B) with radius BD, and a third at f(C) with radius CD. The intersection of these three circles must be f(D).
This means that if there are any three non-colinear points in the square that stay in the same place, then every point has to stay in the same place.
Who says it isn't? In the quoted wiki article we read:
Most generally the symmetry group of an object is just a set of all auto isomorphisms of that object. The word "relevant" is crucial here and in different context it means different things.
If you look at the square as a set then isomorphism = bijection and your example is valid. If you look at it as a topological space then isomorphism = homeomorphism and your example fails cause switching 2 points is not continuous. If you look at it as a subset of some vector space then isomorphism may mean linear isomorphism. If it is metric space then it may mean isometry and so on and so on...