Given a polynomial $f\in\mathbb{C}[x]$ and a point $x_0\in\mathbb{C}$ I am trying to decide if the set $\{x_0,x_1=f(x_0),x_2=f(x_1),\cdots\}$ is finite or infinite.
My general strategy is to iterate $f$ until the absolute value is large enough that the orbit must be infinite, or until I detect a loop.
So far I have been able to prove that once the value exceeds $k/|a_n|$ the orbit must be infinite, where $a_n$ is the leading coefficient and k is the sum of the absolute values of the other coefficients. (Where this is not well-defined the problem is easy.)
So my questions are: is the general strategy a good one? Can my general bound be improved -- indeed, is it correct?
I would be interested also in the restriction to integer polynomials and values.
1. Height bounds of preperiodic points
If the set is finite, then $x_0$ is a preperoidic point so that $$f^{n+m}(x_0)=f^m(x_0)$$ for some $m,n\in \Bbb Z$. Then $f^m(x_0)$ is a periodic point.
Northcott's theorem says that if $f$ is rational and has degree $\geq 2$, then the number of algebraic preperiodic points is finite. Therefore the height strategy should always work for your polynomial case, or you can use any type of counting function. The bound computation is typically done via the equation $$h(f(\beta))=d\cdot h(\beta)+C,$$ where $h$ is the absolute logarithmic height $$h\left(\frac{a}{b}\right)=\log(\max\{|a|,|b|\}) \text{ for }\frac{a}{b}\in\Bbb Q$$ and $C$ is a constant dependent only on $f$. So once $h(\beta)$ is sufficiently large, its iterates grows unbounded. i.e. when $h(\beta)> |C|$: $$h(f(\beta))-h(\beta)=d\cdot h(\beta)-h(\beta)+C\geq 2h(\beta)-h(\beta)+C=h(\beta)+C>0$$ so that it cannot be a preperiodic point.
It sounds like what you are describing to find the bound is similar, since it depends only on $f$. I am not so sure what happens when $x_0$ is non-algebraic, but I guess you can try to derive a similar equation for some cases, possibly using some variant of $h$. I have no idea what happens for other types of functions $f$.
2. Reduction mod $p$
For periodic points, you can consider reduction mod $p$ when it applies. For example, suppose $f\in \Bbb Q(z)$, then we have the following:
Which implies that we only have to test at most $p^3-p$ times. Furthermore, $f$ must have good reduction somewhere so such $n$ can always be found. This result can also be extended to Number fields, so that you can consider $f\in \overline{\Bbb Q}(z)$ and $x_0\in \overline {\Bbb Q}$.
However, to check for preperiodic points one must estimate the maximum value of $m$ as above. I seem to recall some results for this, but unfortunately I have forgotten. You might want to ask for references in Mathoverflow if you are interested.
Otherwise, this trivially applies for $f\in\Bbb Z[x]$ and $x_0\in \Bbb Z$.
3. Linear conjugation
Let $g(z)=\dfrac{az+b}{cz+d}\in \Bbb C(z)$ and define $$f^g(z)=(g^{-1}\circ f\circ g)(z)$$ Then we can show that $$(f^g)^n=(g^{-1}\circ f\circ g)^n=g^{-1}\circ f^n\circ g=(f^n)^g$$ Now if $x_0$ is a preperiodic point for $f$ of period $n$, then $g^{-1}(x_0)$ is a preperiodic point for $f^g$ of period $n$.
Let $g(z)=z+x_0$. Then testing whether $x_0$ is preperiodic for $f$ is the same as testing whether $$g^{-1}(x_0)=0$$ is preperiodic for $$(f^g)(z)=f(z+x_0)-x_0.$$ This may make it easier for some analysis. For instance, if you know all preperiodic points for $f$ (fixed) then you know all preperiodic points for any $f^g$.
4. Integral points
Lastly, if you are considering number of integers in orbit, then it is known to be finite when $f\in \Bbb Q(x)$ and $x_0\in \Bbb Q$. i.e. let $\varphi(x_0)=\{f^i(x_0): i\geq 0\}$ then $$\varphi(x_0)\cap\Bbb Z<\infty$$