EDIT: Solved, it was an algebra mistake. I posted the correct answer below.
I'm interested in explicit formulas for the action of the isometries of the Mobius group on the upper-half space model of $\mathbb{H}^3$. I've seen that the action on the boundary $\partial \mathbb{H}^3 = \hat{\mathbb{C}} = \mathbb{C} \cup\{\infty\}$ is exactly the Mobius transformations.
Below I spell out my attempt to derive the action, although it doesn't work. Any explicit formulas or references would be greatly appreciated, as would an indication of why the method I use below shouldn't work would be appreciated, since it seems to indicate some trickiness with intuition of hyperbolic geometry (or a silly algebra mistake I couldn't catch after much time spent debugging).
Given the metric $ds^2 = \frac{1}{r^2}(dx^2 + dy^2 + dr^2)$ and setting $z = x + iy$, it's clear that transformations the form $z \mapsto (z + b)$ for $b = x' + i y'$ act as translations $\{x \mapsto x + x', y \mapsto y + y', r \mapsto r\}$. And similarly, rotations of the form $z \mapsto a z$ for unit complex numbers $a = e^{i \theta}$ act as rotations of $(x,y) \mapsto (\cos(\theta) x - \sin(\theta) y, \sin(\theta) x + \cos(\theta)y)$ and $r \mapsto r$. And scalings of the form $z \mapsto \lambda z$ for $\lambda \in \mathbb{R}^+$ act as $(x,y,r) \mapsto (\lambda x,\lambda y,\lambda r)$, which preserve the metric.
So the remaining question is how the map $f: z \mapsto 1/z$ looks, since all Mobius transformations can be generated by this and the actions above. My intuition was that since geodesics should map to geodesics and geodesics are circles that are perpendicular to the boundary $\partial \mathbb{H}^3$, one should be able to derive the action of $f(x,y,r)$ by the following procedure.
- Write down two geodesic circles $C_1,C_2$ passing through $(x,y,r)$.
- Say that $C_1$ and $C_2$ intersect $\partial \mathbb{H}^3$ at $z_1,z_2$ and $z_3,z_4$ respectively.
- Then the point where the circles $C_1',C_2'$ defined by $\frac{1}{z_1},\frac{1}{z_2}$ and $\frac{1}{z_3},\frac{1}{z_4}$ intersect will be $f(x,y,r)$
I carry out this procedure first for the case for the case $(x,0,r)$ where $z$ is on the positive real axis in the below images.
Choosing $z_1 = x-r, z_2 = x+r$ and $z_3 = x-ir, z_4 = x+ir$, the above procedure yields that the circles $C'_1,C'_2$ intersect at $\left(\frac{x}{x^2+y^2+r^2},0,\frac{r}{x^2+r^2}\right)$.
This gives a candidate function $$\left(\frac{x}{x^2+r^2},0,\frac{r}{x^2+r^2}\right)$$ for points above the positive real axis. And conjugating by a rotation bringing a point to the positive real axis would yield $$\left(\frac{x}{x^2+y^2+r^2},\frac{-y}{x^2+y^2+r^2},\frac{r}{x^2+y^2+r^2}\right)$$
And, one can check that $f(f(x,y,r)) = (x,y,r)$, which would be expected since $1/(1/z) = z$. And also, the above transformation does indeed correspond to an isometry of the metric.
The best extension formula I know uses quaternions. Identify the elements of the upper half-space with quaternions of the form $$ z= u+ vj, u\in {\mathbb C}, v>0. $$ Then linear-fractional transformations $\gamma\in PSL(2, {\mathbb C})$ given by $$ \gamma(u)= (au+b)(cu+d)^{-1}, ad-bc=1, $$ extend to isometries of the upper half-space by the formula $$ \gamma(z)= (az+b)(cz+d)^{-1}. $$ See section 2.1 in
Ahlfors, Lars V., Moebius transformations in several dimensions, Ordway Professorship Lectures in Mathematics. Minneapolis, Minnesota: University of Minnesota, School of Mathematics. 150 p. (1981). ZBL0517.30001.
This is very cute, but I am unaware of any uses of this extension formula.