Suppose that $a,b$ and $c$ are nonzero real numbers. Define $$h(x) = \frac{ax+b}{bx+c}$$ for $x\neq -\frac cb$. Determine all triples $(a,b,c)$ for which $h(h(x)) =x$ for every real number $x\neq -\tfrac cb$, and $h(x)\neq-\frac cb$.
To start off, I don't know what it means to define $h(x)$, so I just tried solving.
$$h(x)=h^{-1}(x)$$ $$\frac{ax+b}{bx+c}=\frac{b-cx}{xb-a}$$
From this form I can see that $$a=-c$$ but I do not know how to find $b$ or how to find all triples $(a,b,c)$.
How would I solve this problem. Thanks.
Here's another approach. First write out what $h(h(x))$ is $$h(h(x))=h\left(\frac{ax+b}{bx+c}\right)=\frac{a\left(\tfrac{ax+b}{bx+c}\right)+b}{b\left(\frac{ax+b}{bx+c}\right)+c}=\frac{a(ax+b)+b(bx+c)}{b(ax+b)+c(bx+c)}.$$ Next set this equal to $x$ and clear out the numerator to get $$a(ax+b)+b(bx+c)=x(b(ax+b)+c(bx+c))=b(a+c)x^2+(b^2+c^2)x,$$ and moving everything to one side then yields a quadratic equation in $x$ $$b(a+c)x^2+(c^2-a^2)x-b(a+c)=0,$$ which must hold for all $x$. This only happens when all coefficients are $0$, so we must have $$b(a+c)=0\qquad\text{ and }\qquad c^2-a^2=0.$$ This is true if and only if $a=-c$, or $a=c$ and $b=0$. The latter cannot occur as $a,b,c\in\Bbb{R}$ must all be non-zero, so the triplets $(a,b,c)$ for which $h(h(x))=x$ holds are precisely the triplets for which $c=-a$.