$P$ is the set of all people. $$R=\{ ( x,y )\in P \times P \mid y \text{ is the father of x}\}$$
Then, from the definition of composite,
$$R\circ R=\{ ( a,c ) \in P\times P\mid ( \exists b\in P ) ( ( a,b) \in R\wedge ( b,c )\in R ) \}$$
Which I think can be translated to: if $a$ is a child and it has some father, $b$, and the father has a father, $c$, then the child and "grandfather" are contained in $R \circ R$. However, the book says this is not the answer. The composite is not "one is a grandfather to the child". Why? Where did my thinking go wrong, or my understanding of composite?
"Grandfather" is not restrictive enough to describe the relationship between $a$ and $c$ if $(a,c)\in R\circ R$. The English term you're after is "paternal grandfather", which specifies that we're talking about a father's father, and not a mother's father.
Your reasoning looks sound, though. I don't see anything wrong apart from the use of the unqualified "grandfather".