Given $f(x) = \frac1{ax+b}$, for which $a$, $b$ such that $x_1=f(x_3) $, $ x_2=f(x_1) $, $x_3=f(x_2) $ are distinctive

Question

For real numbers $a$ and $b$ define

$$f(x) = \frac1{ax+b}$$

For which $a$ and $b$ are there three distinct real numbers $x_1$, $x_2$, $x_3$ such that $f(x_1) = x_2$, $f(x_2) = x_3$ and $f(x_3) = x_1$?

I tried to isolate a/b, and I found 3 values: f(x1) = x2 ---> 1/(a.x1 + b) = x2 ---> a.x1.x2 + b.x2 = 1 ---> I f(x2) = x3 ---> 1/(a.x2 + b) = x3 ---> a.x2.x3 + b.x3 = 1 ---> II f(x3) = x1 ---> 1/(a.x3 + b) = x1 ---> a.x1.x3 + b.x1 = 1 ---> III make I=II then, a/b = (x3 - x2)/x2.(x1 - x3) ---> x1 ≠ x3, for example, but i don't know how to proceed from here.

Answer 1

Assume $a\ne 0$. Then, $$x_1 = \frac1{ax_3+b}= \frac1{a\frac1{ax_2+b}+b}=\frac1{a\frac1{a\frac1{ax_1+b}+b}+b}$$

which leads to

$$(a+b^2)(ax_1^2+bx_1-1) = 0\tag 1$$

and similarly,

$$(a+b^2)(ax_2^2+bx_2-1) = 0,\>\>\>\>\>(a+b^2)(ax_3^2+bx_3-1) = 0\tag 2$$

If $a+b^2\ne 0$, then the solutions are

$$x_1 = x_2=x_3=\frac{-b\pm\sqrt{b^2+4a}}{2a}$$

and there are no real values of $a$ and $b$ that would make the three $x$'s distinct. On the other hand, if $a+b^2=0$, there are infinite number of solutions to (1) and (2) which can all be distinct. Thus, the values of $a$ and $b$ satisfy

$$a+b^2=0$$

Answer 2

Here is a linear algebra approach. It is long because I am trying to explain what is going on. However, the approach of pairing a map of the form $f(x)=\frac{ax+b}{cx+d}$ with the matrix $A=\begin{pmatrix}a&b\\c&d\end{pmatrix}$ is very useful. For example, if $f(x)=\frac{ax+b}{cx+d}$ and $\tilde{f}(x)= \frac{\tilde{a}x+\tilde{b}}{\tilde{c}x+\tilde{d}}$ correspond to the matrices $A=\begin{pmatrix}a&b\\c&d\end{pmatrix}$ and $\tilde{A}=\begin{pmatrix}\tilde{a}&\tilde{b}\\\tilde{c}&\tilde{d}\end{pmatrix}$, respectively, then $f\circ \tilde{f}$ corresponds to the matrix $A\tilde{A}$.

Back to the question, recall that $f(x)=\frac{1}{ax+b}=\frac{0x+1}{ax+b}$. First of all $a\ne 0$ (otherwise $f$ is constant which obviously doesn't fit the bill). Let $A$ be the matrix $\begin{pmatrix}0&1\\a&b\end{pmatrix}$. The fact that $a\ne 0$ means that $A$ has rank $2$. Therefore $A$ is invertible (whence the eigenvalues of $A$ are non-zero).

Associate a real number $x$ with any non-zero vector $\xi=\begin{pmatrix}u\\v\end{pmatrix}$ s.t. $v\ne 0$ and $\frac{u}{v}=x$. We use the notation $x\sim\xi$ in this case. Also associate $\infty$ with vectors $\zeta=\begin{pmatrix}w\\0\end{pmatrix}$ with $w\ne 0$. (Using $\infty$ is useful here. We can for example set $f(-b/a)=\infty$ and $f(\infty)=0$.) Observe that if real numbers $x$ and $y$ satisfy $x\sim\xi$ and $y\sim \xi$ for some non-zero vector $\xi$, then $x=y$.

Suppose that $x\sim \xi$. Then show that $f(x) \sim A\xi$. Now let $x_1$, $x_2$, and $x_3$ be distinct real numbers s.t. $f(x_1)=x_2$, $f(x_2)=x_3$, and $f(x_3)=x_1$. If $x_i\sim \xi_i$ for $i=1,2,3$, then $$x_2=f(x_1)\sim A\xi_1.$$ But we have $x_2\sim \xi_2$ also. By the definition of $\sim$, this means $A\xi_1=r\xi_2$ for some non-zero real number $r$. Now $$x_3=f(x_2)\sim A\xi_2$$ and $x_3\sim \xi_3$. Using the same argument, $A\xi_2=s\xi_3$ for some non-zero real number $s$. Finally from $$x_1=f(x_3)\sim A\xi_3$$ and $x_1\sim \xi_1$, we get that $A\xi_3\sim t\xi_1$ for some non-zero real number $t$.

Therefore $$A^3\xi_1=A^2(A\xi_1)=A^2(r\xi_2)=rA^2\xi_2=rA(A\xi_2)=rA(s\xi_3)=rs(A\xi_3)=rst\xi_1.$$ Therefore $A^3$ has a (non-zero) real eigenvalue $rst$. Since $A$ is a real $2\times 2$ matrix, either it has two real eigenvalues or two non-real conjugate eigenvalues. We want to prove that $A$ doesn't have a real eigenvalue.

Suppose for the sake of contradiction that both eigenvalues of $A$ are real numbers $\lambda_1,\lambda_2$. Note that $$\det(A^2+kA+k^2I)=(\lambda_1^2+k\lambda_1+k^2)(\lambda_2^2+k\lambda_2+k^2)>0$$ if $k=\sqrt[3]{rst}$. Hence $(A^2+aA+a^2I)$ is invertible.

Now $$(A^2+kA+k^2I)(A-kI)\xi_1=(A^3-k^3I)\xi_1=A^3\xi_1-k^3\xi_1=A^3\xi_1-rst\xi_1=0.$$ So $$A\xi_1-k\xi_1=(A-kI)\xi_1=(A^2+kA+k^2I)^{-1}0=0.$$ That is $A\xi_1=k\xi_1$. But we know $A\xi_1=r\xi_2$. Therefore $k\xi_1=r\xi_2$. That means $x_1$ and $x_2$ are associated to $\xi_2$. This shows that $x_1=x_2$, contradicting the assumption that $x_1$, $x_2$, and $x_3$ are distinct. Hence the eigenvalues of $A$ must be non-real complex numbers $\lambda$ and $\bar{\lambda}$.

Now $rst$ is an eigenvalue of $A^3$, so we must have $\lambda^3=k^3$ or $\bar{\lambda}^3=k^3$ (recalling that $k^3=rst$ is real). In the latter case, $\lambda^3=\overline{\left(\bar{\lambda}^3\right)}=\overline{k^3}=k^3$, so we still have $\lambda^3=k^3$ anyhow. Therefore, $\lambda$ is an eigenvalue of $A$ which is non-real but $\lambda^3$ is the real number $k^3$. Since $\lambda$ is non-real, $\lambda\ne k$, so $$\lambda^2+k\lambda+k^2=\frac{\lambda^3-k^3}{\lambda-k}=0.$$ This shows that the characteristic polynomial of $A$ is $$\det(A-qI)=q^2+kq+k^2.$$ But $$\det(A-qI)=\det\begin{pmatrix}-q&1\\a&b-q\end{pmatrix}=q^2-bq-a.$$ Therefore $-b=k$ and $-a=k^2$, so $$a=-k^2=-(-k)^2=-b^2.$$ And we recall that $a\ne 0$, so the necessary condition is $a=-b^2\ne 0$. This condition is also sufficient because when $a=-b^2\ne0$, $$A^3=-b^3I.$$ This means that $f\circ f\circ f$ is the identity. Since $A$ has no real eigenvalue, $f$ has no fixed point. Consequently, for any $x\in \Bbb R\cup\{\infty\}$, the values of $x$, $f(x)$, and $f\circ f(x)$ are distinct, and $f\circ f\circ f(x)=x$.