This is a piece of folklore I found on the Internet:
$$ \begin{array}{ll} \varepsilon = x & f = 1 - x \\ g = \frac{1}{x} & h = \frac{1}{1 - x} \\ m = \frac{x - 1}{x} & n = \frac{x}{x - 1} \end{array} $$
There are 6 functions, and it is claimed that they form the dihedral group $D_3$ under composition. I computed some part of the corresponding (hypothetical yet) Cayley table, and it does look like a group. However, the derivations, trivial as they may be, take a lot of paper and attention to actually perform, and a slight mistake will lead one to believe that this is not a group. What kind of tricks can I use to find out if this is a group, short of carefully computing the whole table?
It appears that all the functions here are distinct, but technically I should provide for every pair of functions a value that differentiates them. For example, $\varepsilon(2) = 2, h(2) = -1$, and so on. This is really so boring. As a shortcut, I may compare limits and observe:
$$ \begin{array}{ll} \varepsilon \to +\infty & f \to -\infty \\ g \to 0^+ & h \to 0^- \\ m \to 1^- & n \to 1^+ \end{array} $$
So far so good.
$\varepsilon$ is the identity.
$g$ is the multiplicative inverse, so in the group $\langle\mathbb{R}, \times\rangle$ $g \circ g$ would be $(a^{-1})^{-1} = a$, so $g \circ g$ = $\varepsilon$ and we have our first subgroup $Z_2$.
This is about as far as I could get, until I decided to explore the compositional structure of the given functions. Turns out that I can express them all in terms of these trivial functions:
$$ \begin{array}{rl} \alpha & = & x + 1 \\ \beta & = & -x \\ \gamma & = & \frac{1}{x} \end{array} $$
$$ \begin{array}{rl} & f = \alpha\beta \\ g = \gamma & h = \gamma\alpha\beta \\ m = \alpha\beta\gamma & n = \alpha\beta\gamma\alpha\beta \end{array} $$
Only a handful of simple theorems are enough to infer the whole Cayley table, giving a witness to group structure:
$$ \beta\beta = \gamma\gamma = \varepsilon \tag{involution} $$ $$ \beta\alpha\beta = \alpha^- \tag{inverse} $$
It took only a page to get there. (Identity omitted for clarity.)
$$ \begin{array}{c|cccccc} & f & g & h & m & n \\ \hline f & & m & n & g & h \\ g & h & & f & n & m \\ h & g & n & m & & f \\ m & n & f & & h & g \\ n & m & h & g & f & \\ \end{array} $$
I wonder though if there is a yet simpler way. The given functions are so neat, they are obviously specially prepared. But I am not seeing how. For example, would it be difficult to offer a collection of 8 functions that form the group $D_4$ of the symmetries of the square under composition? Is there some generality I am not seeing?
To any nonzero point $(x, y)$ in $\mathbb{R}^2$, we can associate a (possibly infinite) slope $s = \frac{y}{x}$. Suppose that under a linear transformation $T$, the point $(x, y)$ is sent to the new point $(ax + by, cx + dy)$. Then the slope of this new point is equal to $$\frac{cx + dy}{ax + by} = \frac{ds + c}{bs + a},$$ so we see that linear transformations on $\mathbb{R}^2$ induce so-called "fractional linear transformations" on the slopes of points. The map $\varphi$ taking a linear transformation to its induced fractional linear transformation is a group homomorphism from the group of invertible linear transformations on $\mathbb{R}^2$ (aka GL$_2(\mathbb{R})$) to the group of invertible fractional linear transformations (for both groups the group multiplication is composition). You can straightforwardly compute that the kernel of $\varphi$ is the group of dilations of the plane, so if $G$ is a subgroup of GL$_2(\mathbb{R})$ that doesn't contain any dilations, then $\varphi(G)$ is a collection of fractional linear transformations that is isomorphic to $G$.
For your particular collection of fractional linear transformations, consider the three points $(0,1)$, $(1,1)$, and $(1,0)$ in the plane. You can compute that the subgroup $G$ of GL$_2(\mathbb{R})$ that stabilizes these points is isomorphic to $D_3$, that $G$ doesn't contain any dilations, and that $\varphi(G)$ is exactly the collection of fractional linear transformations that your post is about. To construct $D_n$ for any odd $n$ out of fractional linear transformations, you can instead start with the stabilizer in GL$_2(\mathbb{R})$ of the vertices of a regular $n$-gon, and apply the same procedure. This won't work to construct $D_n$ for even $n$, since one of the symmetries of an even $n$-gon is 180-degree rotation, which is the same thing as a dilation by a factor of -1. Perhaps there's a more clever geometric construction that one can do.