Let $f^n(x)$ denote the nth composition of $f(x)=\dfrac{ax+b}{bx+a}$. Find an explicit expression for $f^n(x) $.
Insofar, I noticed on taking the first 3 compositions of $f(x)$, the terms from the binomial expansion of $(a+b)^n$ seem to pop up, but I can't find a precise pattern or a way of proving this assertion via induction;
$f^2(x)=\dfrac{(a^2+b^2)x+2ab}{2abx+(a^2+b^2)}$
$f^3(x)=\dfrac{(a^3+3ab^2)x+(3a^2b+b^3)}{(b^3+3a^2b)x+(a^3+3ab^2)}$
$f^4(x)=\dfrac{(a^4+b^4+6a^2b^2)x+(4a^2b+4ab^2)}{(4a^2b+4ab^2)x+(a^4+b^4+6a^2b^2)}$
The nth composition of $f(x)$ will have the form $f(x)=\dfrac{Ax+B}{Bx+A}$ where A and B will contain the binomial coefficients of $(a+b)^n$ but how can I compute $A$ and $B$ here?
$f^n(x)=\dfrac{A_n x+B_n}{B_n x+A_n},$then $f(f^n(x))=\dfrac{(aA_n+b B_n)x+aB_n+bA_n}{(aB_n+bA_n)x+aA_n+B_n}.$
$P_n=\left(\begin{matrix}A_n & B_n \\ B_n & A_n\end{matrix}\right),P_{n+1}=P_n\left(\begin{matrix}a & b \\ b & a\end{matrix}\right).$Thus $P_n=\left(\begin{matrix}a & b \\ b & a\end{matrix}\right)^n$.
Consider $A=\left(\begin{matrix}a & b \\ b & a\end{matrix}\right)$. $|\lambda I-A|=(\lambda-a+b)(\lambda-a-b).\implies P^{-1}AP=\left(\begin{matrix}a-b & 0 \\ 0 & a+b\end{matrix}\right).$Here $P=\left(\begin{matrix}1 & 1 \\ -1 & 1\end{matrix}\right).$
Hence $P_n=A^n=P\left(\begin{matrix}(a-b)^n & 0 \\ 0 & (a+b)^n\end{matrix}\right)P^{-1}=\left(\begin{matrix}(a-b)^n/2+(a+b)^n/2 & -(a-b)^n/2+(a+b)^n/2 \\ -(a-b)^n/2+(a+b)^n/2 & (a-b)^n/2+(a+b)^n/2\end{matrix}\right).$