Let $R$ denote an etale equivalence relation on a scheme $X \in \operatorname{Sch}/S$.
The statement of Proposition 5.2.5 in Olsson's Algebraic spaces and stacks is as follows:
(i) $X/R$ is an algebraic space, where $X$ is the associated sheaf with respect to the etale topology of the functor $(\operatorname{Sch}/S)^{op} \to \operatorname{Set}, T \mapsto X(T)/R(T)$
(ii) If $Y$ is an algebraic space over $S$, and $X \to Y$ is an etale surjective morphism, then $R:= X \times_Y X$ is a scheme and the inclusion $R \hookrightarrow X \times_S X$ is an etale equivalence relation.
How is the map $R \to X \times_S X$ defined?
Let $f\colon X \to Y$, $g\colon Z \to Y$, and $h\colon Y \to S$ be morphisms in any category with pullbacks. Then we have a commutative diagram $$\require{AMScd}\begin{CD} X \times_Y Z @>>> Z \\ @VVV @VV{h\circ g}V \\ X @>{h\circ f}>> S \end{CD}$$ where the morphisms from $X \times_Y Z$ are the projection maps (which are part of the data of the pullback). Applying the universal property of the pullback $X \times_S Z$ to the above diagram, the maps $X \times_Y Z \to X$ and $X \times_Y Z \to Z$ factor through a unique map $$X \times_Y Z \to X \times_S Z.$$ This is the map you're asking about (with $X = Z$ in your case).
To get a better sense of what this map looks like in practice, let's consider the case of a concrete category (where we can think of the objects as "sets with some extra structure and properties", and the morphisms as "structure-preserving functions"). The fiber product $X \times_Y Z$ is the subset of $X \times Z$ consisting of pairs $(x, z)$ such that $f(x) = g(z)$. If $f(x) = g(z)$, then certainly $h(f(x)) = h(g(z))$, so the underlying set of $X \times_Y Z$ is a subset of the underlying set of $X \times_S Z$.
The picture is slightly more complicated for schemes since we also have to consider the ringed space structure, but this should at least give some intuition for why this is an inclusion even if we'd need to say a bit more to turn this into a formal proof.