Any action of a finite group on a (non-empty) tree has a global fixed point (in the sense that there is a vertex fixed by all group elements or an edge fixed by all group elements).
There is a hint which says we can consider the diameter of the corresponding orbit is minimal. However I don't find the definition of diameter in the book.
Can someone clarify the diameter? Or give the details of proof? Thank you.
On
Any orbit of a vertex is a collection of vertices. As such, it is a subgraph of the given graph and hence is itself a graph. Thus, for this case, my guess is that "diameter" is in the graph theoretic sense. But what would it mean for edges? I'm not sure. In any case, the orbit is itself a metric space and the diameter of a metric space $(X,d)$, is well-known:
$diam(X,d) = \sup_{x,y \in X} d(x,y).$
Let $G$ be a finite group acting on a tree $T_0$; let $T$ denote its barycentric subdvision, that is add the middlepoint of each edge as a vertex. Thus, the action $G \curvearrowright T_0$ has a global fixed point (in the sense you mention) iff the action $G \curvearrowright T$ fixes a vertex.
Solution 1: Let $Y \subset T$ be a bounded set. Its radius is defined as $$r_Y= \inf \{ r>0 \mid \exists x, \ Y \subset B(x,r) \}.$$ A point $x \in X$ is a center of $Y$ if $Y \subset B(x,r_Y)$.
A fundamental property is that any bounded set in a tree has a unique center. Therefore, if $Y$ is a bounded orbit (in your case, any orbit works since $G$ is finite), and $x$ is its center, then for all $g \in G$, $g \cdot x$ is the center of $g \cdot Y = Y$ hence $g \cdot x=x$. That is, $x$ is fixed by $G$ (and it is necessarily a vertex of $T$).
In fact, the proof works in any CAT(0) space.
Solution 2: Using Bass-Serre theory, and because any HNN extension or nontrivial free amalgamation is infinite, $G$ may be written as a trivial free amalgamation $A \underset{C}{\ast} B$, ie. $G=A$ or $B$. Because $A$ and $B$ are identified with the stabilizer of some vertex, we deduce that $G$ fixes a vertex.