Let $f(x)=x^2-x-2$. It is given that $a, b,c \in \mathbb{R}$ such that $$f(a)=b,f(b)=c,f(c)=a$$ Then find the value of $\frac{1}{a}+\frac{1}{b}+\frac{1}{c}$.
My attempt: We have: $$\begin{array}{l} a^{2}-a-2=b \\ b^{2}-b-2=c \\ c^{2}-c-2=a \end{array}$$ Subtracting pair wise we get: $$(a-b)(a+b)=a-c$$ $$(b-c)(b+c)=b-a$$ $$(c-a)(c+a)=c-b$$ Trivially $a=b=c$ satisfies. So we get $a=b=c=\sqrt{3}+1, 1-\sqrt{3}$ Thus we get $$\frac{1}{a}+\frac{1}{b}+\frac{1}{c}=\frac{3\sqrt{3}+3}{2}, \frac{-3-3\sqrt{3}}{2}$$ Now if $a\ne b \ne c$ Then we have: $$(a+b)(b+c)(c+a)=-1$$
Letting $p=a+b+c$ we have $$(p-a)(p-b)(p-c)=-1$$ $\implies$ $$p(ab+bc+ca)-abc+1=0$$
Any hint here?
From the condition, we need $f\circ f\circ f(a)=a$. Now $$f(x) = x^2-x-2=(x-2)(x+1)$$ $$f\circ f(x) = (x^2-x-4)(x^2-x-1)=x^4-2x^3-4x^2+5x+4$$ $$\implies f\circ f\circ f(x)=(x^4-2x^3-4x^2+4x+2)(x^4-2x^3-4x^2+4x+5)$$ $$\implies f\circ f\circ f(a)-a=a^8-4a^7-4a^6+26a^5+3a^4-54a^3-3a^2+34a+10 \\=(a^2-2a-2)(a^3-3a-1)(a^3-2a^2-3a+5)$$
It is easy to note all roots are real (using Descartes and evaluation at $a=-1, 0, 1$), so we have $8$ possibilities for $a$, which fall into two cases.
Case 1: The first/quadratic factor gives $a=1\pm\sqrt3$, each of which is also a solution to $f(a)=a$, so as you have already noted, we have two values for $\frac1a+\frac1b+\frac1c=\frac3a$ from these, viz. $\frac12(\pm3\sqrt3-3)$.
Case 2: Now we have no more roots of $f(a)=a$, hence the rest of the roots (six roots from the two cubic factors), must contribute to two possible sets of $\{a, b, c\}$.
Taking the first cubic factor, $\alpha^3-3\alpha-1=0 \implies f(\alpha)^3-3f(\alpha)-1 = (\alpha^2-\alpha-2)^3-3(\alpha^2-\alpha-2)-1 \\=(\alpha^3-3\alpha-1)(\alpha^3-3\alpha^2+3)=0$
i.e. if $\alpha$ is a root of the first cubic, so is $f(\alpha)$. Thus the roots of that cubic contribute one set of $\{a, b, c\}$. Now using Vieta, sum of the reciprocal of roots is $$\frac1a+\frac1b+\frac1c=\frac{ab+bc+ca}{abc}=-3$$
Similarly the roots of the last cubic form the last set of possible $a, b, c$, hence have a sum or reciprocals $=\frac35$.
Summarising the value of $\frac1a+\frac1b+\frac1c \in \{\frac12(\pm3\sqrt3-3), -3, \frac35\}$, four possibilities.