I understand that permutationgroups in Gap are represented by generators, which seems to be far more efficient than groups represented by all it's elements, but how could then for example
gap>Elements( Group( (1,2,3,4,5,6,7,8),(1,2) ) );
so fast list all $8!$ elements of $S_8$? Can someone describe the algorithm or the method used?
Alex's answer is very good, but I'd like to add a somewhat simplified explanation of the Schreier-Sims Algorithm, since @Lehs asked for it.
I want to emphasis that this is not the actual Schreier-Sims algorithm but it contains the essential ideas in the Schreier-Sims algorithm, and I think it will serve to illustrate why it is preferable to the brute force exhaustive algorithm.
I will try to describe this not the Schreier-Sims algorithm using an example.
Let $G$ be the group generated by the permutations: $$a=(2\ 3),\quad b=(1\ 2\ 3)(4\ 5).$$ We will use not the Schreier-Sims Algorithm to find the size of this group. This is a bit simpler than finding the actual elements of the group.
We require two concepts:
By the Orbit-Stabilizer Theorem, $|G| = |\alpha \cdot G|\cdot |G_{\alpha}|$, i.e. the size of $G$ is the size of the orbit of $\alpha$ multiplied by the size of the stabiliser of $\alpha$. This is also important, and we will use this later.
But how do we calculate the stabiliser? One way (bad!) would be to enumerate all of the elements of $G$ and check which elements fix $1$. Another way (good!) is to use the following lemma:
Schreier's Lemma. If $G$ is generated by a set $X$ and $G$ acts on a set $\Omega$, then $$G_\alpha = \langle u_ixu_j^{-1} : 1\leq i, j\leq n, x\in X, \alpha_i \cdot x = \alpha_j\rangle$$ where $\alpha\cdot G = \{\alpha_1, \alpha_2, \ldots, \alpha_n\}$.
In other words, if you keep track of what maps what to what when enumerating the orbit of $\alpha$, then you get a generating set for the stabiliser for free (more or less).
Returning to our example $$ 1\cdot G = \{1, 2, 3\}\quad G_1 = \langle (2\ 3), (4\ 5)\rangle.$$ Note that it is possible to find the generators for the stabiliser $G_1$ using Schreier's Lemma, but this is a bit tedious so I have not given any of the details. We repeat this process on the stabiliser $G_1$: $$ 2\cdot G_1 = \{2, 3\} \quad G_{1, 2} = \langle (4\ 5)\rangle$$ where $G_{1,2}$ means the stabiliser of $2$ in $G_1$ (again omitting the details). We repeat this process again on $G_{1,2}$: $$4\cdot G_{1,2} = \{4, 5\} \quad G_{1, 2, 4} = \{\text{id} \}.$$ Here is where our algorithm ends, by the Orbit-Stabilizer Theorem (applied 3 times to $G$, then $G_1$, then $G_{1,2}$): $$|G| = |1\cdot G|\cdot |G_1| = |1\cdot G| \cdot |2\cdot G| \cdot |G_{1,2}| = |1\cdot G |\cdot |2\cdot G| \cdot |4\cdot G| \cdot |G_{1,2,4}| = 3\cdot 2 \cdot 2 \cdot 1 = 12.$$ The set of initial points in the orbits $\{1, 2, 5\}$ is often called a base, the union of the generators of $G$ with the generators of the stabilisers $G_1$, $G_{1,2}$, and $G_{1,2,4}$ is called strong generating set, and the orbits and stabilisers a called a stabiliser chain. You can use a stabiliser chain to answer more questions than just finding the size, for example, it can be used to check membership in the group $G$ and it can be used to find the elements in $G$.
In this rather simple example, to do the exhaustive algorithm (to find the size) is not much more difficult than the algorithm proposed above. But think about the example of, say, the symmetric group on 10 points, forgetting that we know its size already. You only have to compute 10 orbits of length 10 to 1 (that 55 points in total). In the absolute worst case (which won't happen in practice), finding the generators of the stabiliser of a point whose orbit is of length $n$ involves $2n$ multiplications of permutations producing (again in the worst case, which doesn't occur) $n$ generators for the stabiliser. So in total we require 350 applications of permutations to points (to calculate the orbits) and 110 multiplications of permutations (to find the generators in Schreier's Lemma). From this you get that the size of this group is $10! = 3628800$.
Compare this to the exhaustive algorithm where we'd require $7257600$ products of permutations on 10 points (never mind anything else).