In my textbook the following functions are given and are said to form a group under composition:
$f_1(x) = x$, $f_2(x) = \frac{1}{x}$, $f_3(x) = 1-x$, $f_4(x) = \frac{1}{1-x}$, $f_5(x) = \frac{x-1}{x}$, $f_6(x) = \frac{x}{x-1}$
As I understand it, composition means simplifying all possibilities of $f_n(f_m(x))$, and then testing for the group checks.
I understand that they form a group due to the fact that they satisfy the four axioms (closure, associativity, identity, inverse), but wish to understand exactly what about the functions causes this.
The best way I can put this is, if you needed to create another set, where would you start? I assume $f_1(x)$ is the only possible Identity Element for composition of functions, is there something else about the given functions that cause them to form a group?
As of now, this is all I notice:
$f_1(x)^{-1} = f_2(x), f_3(x)^{-1}=f_4(x),f_5(x)^{-1}=f_6(x)$
This is a famous example of a finite group of linear fractional transformations. These are transformations of the form $$ x\mapsto \frac{ax+b}{cx+d}, $$ wher $ad-bc\neq0$. The use of the term "linear transformation" here comes from the following. Imagine the set of lines in the plane through the origin. Such a line $L$ is a 1-dimensional subspace, and thus generated by a single vector, say the vector $\vec{v}_L=(x_L,y_L)$. Because any non-zero scalar multiple of $\vec{v}_L$ generates the same subspace we can scale the $y$-coordinate to be equal to $1$, i.e. use $(x_L/y_L,1)$ as the generator instead. This fails only, when $y_L=0$ (i.e. when the line is horizontal) in which case we scale $x_L$ to $1$ instead.
So the ratio $\lambda_L=x_L/y_L$ determines the line uniquely, and we can cover all the lines, if we define $\lambda=1/0=\infty$ for the horizontal line. The parameter $\lambda$ is thus the reciprocal of the slope of the line $L$.
Anyway, an invertible linear transformation $T$ from the plane to itself will map such a 1-dimensional subspace to another. Assume that the linear transformation amounts to multiplication (from the left side) by the matrix $$ A=\left(\begin{array}{cc}a&b\\c&d\end{array}\right). $$ Let's check what this does to the parameter $\lambda$. We see that $$T(\vec{v}_L)=A\left(\begin{array}{c}\lambda\\1\end{array}\right)=\left(\begin{array}{c}a\lambda+b\\c\lambda+d\end{array}\right).$$
The $\lambda$-parameter of the image $T(L)$ is thus $(a\lambda+b)/(c\lambda+d)$. It is also easy to see that the line with $\lambda=-d/c$ is mapped to the horizontal line with $\lambda=\infty$, and that horizontal line $\lambda=\infty$ is mapped to the line with $\lambda=a/c$ because $T(1,0)=(a,c)$. Getting warmer, right?
Now it is easy to check that composition of planar linear transformations, known to correspond to a product of matrices, also corresponds to the composition of fractional linear transformations. In the language of groups this means that the function mapping the above matrix to the function $\lambda\mapsto (a\lambda+b)/(c\lambda+d)$ is a homomorphism of group.
But, there's a catch. Should $A=aI_2$ be a scalar matrix, that is $a=d, b=c=0$, then $(a\lambda+b)(c\lambda+d)=a\lambda/a=\lambda$ for all $\lambda$. In other words, the scalar matrices map all those lines to themselves. Geometrically this is obvious because the scalar linear transformations simply stretch things radially away from the origin. In the language of groups this means that the scalar matrices belong to the kernel of that homomorphism.
You may have already guessed where this is heading. The functions in your example homomorphic images of the matrices $$ \begin{aligned} A_1&=I_2\mapsto&[x\mapsto x],\\ A_2&=\left(\begin{array}{cc}0&1\\1&0\end{array}\right)\mapsto&[x\mapsto\frac1x],\\ A_3&=\left(\begin{array}{cc}-1&1\\0&1\end{array}\right)\mapsto&[x\mapsto 1-x],\\ A_4&=\left(\begin{array}{cc}0&1\\-1&1\end{array}\right)\mapsto&[x\mapsto\frac1{1-x}],\\ A_5&=\left(\begin{array}{cc}1&-1\\1&0\end{array}\right)\mapsto&[x\mapsto\frac{x-1}{x}],\\ A_6&=\left(\begin{array}{cc}1&0\\1&-1\end{array}\right)\mapsto&[x\mapsto\frac x{x-1}].\\ \end{aligned} $$
The above catch shows here as follows. The above set of six matrices does not form a group under matrix multiplication. For example you easily see that $A_4^2=-A_5$. But, the matrices $A_5$ and $-A_5$ give rise to the same linear fractional transformation. Furthermore, the set of twelve matrices $$G=\{\pm A_i\mid i=1,2,3,4,5,6\}$$ does form a group under matrix multiplications. Leaving it to you to verify that.
In spite of what that Wikipedia article may be implying these groups are very much of interest over fields other than a subfield of the complex numbers, but let's leave that for some other day :-)