I found this question Isomorphisms between group of functions and $S_3$ but they don't show why $\phi$ is an isomorphism.
I found that $$f(x) = 1/x$$ $$g(x) = (x-1)/x$$ $$f(f(x)) = x$$ $$f(g(x)) = x/(x-1)$$ $$g(f(x)) = (1/x - 1)/(1/x) = 1 - x$$ $$g(g(x)) = (((x-1)/x)-1)/((x-1)/x) = ((x-1) -x)/(x-1) = 1/(1-x)$$
The symmetrical group $S_3$ is defined as $$S_3 = (\text{id}, (12), (13), (23), (123), (321))$$
It seems obvious that there are six different possibilities in both cases, and $x$ seems to be equal to $\text{id}$, but how do I show that there is an isomorphism between them ?
First of all, you showed that $f^2 = g^3 = \mathrm{id}$, where powers denote compositions with itself. In consequence, $f$ has to be mapped to an element of order 2 inside $S_3$, that is a transposition $\tau \in S_3$, for instance $\tau = (12)$, and $g$ to a 3-cycle $\sigma \in S_3$, such as $\sigma = (123)$. Those two elements generates $S_3 = \langle \tau,\sigma \,|\, \tau^2 = \sigma^3 = (\tau\sigma)^2 = \mathrm{id} \rangle$, where the identity $(\tau\sigma)^2 = \mathrm{id}$ fixes a "commutation relation" between the two generators; this last relation is also satisfied by the given functions $f$ and $g$.
Now, since every element of a group can be written as a combination of its generators, it is sufficient to consider a homomorphism $\varphi : G \rightarrow S_3$ with $\varphi(f) = \tau$ and $\varphi(g) = \sigma$. If you want a concrete example, let's take $(g \circ f)(x) = 1-x$; it will be mapped to $\varphi(g \circ f) = \varphi(g) \varphi(f) = \sigma\tau = (123)(12) = (13)$.