Let $f,g:\mathbb{R}\rightarrow \mathbb{R}$ be functions given by $f(x)=-x$ and $g(x)=x+1$. They are permutations of $\mathbb{R}$. Let $G$ be the group generated by $f$ and $g$. The group law is composition.
I am wondering whether there is a way to prove that this group $G$ is isomorphic to $H= \langle a,b\mid a^2=e,ab=b^{-1}a \rangle$. I can show that there exists a group homomorphism $\varphi:H\rightarrow G$ but these two groups seems to be both infinite so I need to find an inverse but I don't know how.
Welcome to mse!
First things first, there's no general way to show that two presentations are isomorphic. This is the isomoprhism problem for groups, and it's "well known" to be undecidable in general.
That said, we can solve this particular example!
You've shown there's a group hom $\varphi : H \to G$, presumably by realizing that $f \circ f = \text{id}$ and $f \circ g = g^{-1} \circ f$. Since $f$ and $g$ generate $G$, and are in the image of $\varphi$, this means that $\varphi$ is surjective (do you see why?) so it remains to check that $\varphi$ is injective.
Here's a hint/sketch:
Can you show that every element of $G$ is of the form $f^{i} \circ g^{j}$ where $i \in \{0,1\}$ and $j \in \mathbb{Z}$? The crux of the argument is in showing that anytime we have a $g^n \circ f$, we can replace it by an $f \circ g^{-n}$ so that we end up with all our $f$s to the left of our $g$s.
Once you've done this, can you show that $f^{i} g^{j} = \text{id}$ if and only if $i = j = 0$? Here you'll finally want to use the fact that $f$ and $g$ are functions. If $i$ or $j$ is nonzero, can you show that some real must get moved? Moreover, can you show that $f^i g^j$ and $f^k g^l$ are different functions provided $(i,j) \neq (k, l)$?
Lastly, let's use this to check $\varphi$ is injective! Notice (by a similar argument to what we just did) every element of $H$ is of the form $a^i b^j$ for $i \in \{0,1\}$ and $j \in \mathbb{Z}$. Then $\varphi(a^i b^j) = f^i g^j$, and now it should be easy to check injectivity.
As an aside, this worked by finding a normal form for the elements of $G$ and $H$. This mode of argument is incredibly flexible, and should be on your radar whenever you're doing combinatorics with words.
I hope this helps ^_^