I'm reading the appendix to section III.C of Bridson–Haefliger's Metric Spaces of Nonpositive Curvature on fundamental groups and coverings of small categories. I have two questions about the relationship between the categorical and topological notions of homotopy.
First, let me recall: If $X$ and $Y$ are topological spaces, two maps $f,g\colon X \to Y$ are said to be homotopic if there is a continuous map $F\colon X\times[0,1]\to Y$ with $F(x,0) = f(x)$ and $F(x,1) = g(x)$. If $C$ is a small category (so that there is a set of objects and a set of arrows in $C$), its geometric realization $|C|$ is (I think?) a $\Delta$-complex with vertex set the objects of $C$, edge set the nonidentity arrows of $C$ and an $n$-simplex mapping in for each length-$n$ chain of composable arrows. E.g. if $f$ and $g$ are composable as $fg$ in $C$, there is a $2$-simplex mapping into $|C|$ sending its faces to $g$, $f$, and $fg$, respectively.
Let $C$ and $D$ be small categories, with two functors $F,G\colon C \to D$. The functors are said to be homotopic (p. 575) if, for every object $x \in C$, there is an isomorphism (invertible arrow) $\kappa_x\colon Fx \to Gx$ in $D$ such that if $\gamma\colon x \to y$ is an arrow of $C$, then $G\gamma = \kappa_y\circ F\gamma\circ \kappa_x^{-1}$. It's not hard to see that functors of small categories correspond to simplicial maps between their geometric realizations—although simplices may be crushed.
It's also not hard to see that a homotopy of functors $F,G$ as in the previous paragraph determine a homotopy between the continuous functions $|F|,|G|\colon |C| \to |D|$: the arrows $\kappa_x$ allow you to define a map $|C| \times [0,1] \to |D|$.
Question 1: Why the stipulation that $\kappa_x$ be invertible?
If we soften the requirement to just the existence of arrows $\kappa_x$ as above, but instead require that for each arrow $\gamma\colon x \to y$ in $C$, we have $\kappa_y\circ F\gamma = G\gamma\circ \kappa_x$, then we can still define a homotopy between $|F|$ and $|G|$.
I suspect that the answer to Question 1 has to do with wanting homotopy to be an equivalence relation: if the $\kappa_x$ are not invertible, there is a priori no reason that there should be a homotopy with the roles of $F$ and $G$ reversed, but one could get around this by just asking for the existence in one direction. This latter definition seems to me to more closely follow the definition of homotopy between simplicial (continuous) functions on $\Delta$-complexes.
At the risk of getting circular, Question 2: is there a way to realize homotopy of continuous functions as an instance of homotopy of functors?
I guess what I'm asking for is a functor $\mathscr{F}$ from Top (although I'm happy to accept a restriction to, say, CW complexes) to a category $\mathscr C$ of small categories with arrows functors, such that homotopies of continuous functions $f,g\colon X \to Y$ correspond to homotopies of functors $\mathscr Ff,\mathscr Fg\colon\mathscr FX \to \mathscr FY$? And vice versa? (I'm also happy for the functor to be contravariant if necessary)
For the first question, as noted you don't need the maps $\kappa_x$ to be invertible in $D$. We have a fully faithful embedding of categories into simplicial sets : $$ N : \mathbf{Cat} \to \mathbf{sSet} $$ where $\mathbf{Cat}$ is the category of small categories, and $\mathbf{sSet}$ is the category of simplicial sets. This is called the nerve functor. It is right adjoint to some functor I don't want to describe for the moment, and hence it commutes with small limits.
The thing you described with $\kappa_x$ beeing non invertible is what is called a natural transformation between $F,G$, the data of such a thing is equivalent to the following data : a functor $H:[1]\times C \to D$,where $[1]$ is the category with two objects and one non-isomorphism between them, if you are familiar with the fact that partially ordered sets can be seen as categories, $[1]$ is just the poset $\{0<1\}$. We have to different embeddings of the category $[0]$ (the category with just one object and only the identity morphism) into $[1]$, $\delta_0$ which sends the single object to $0$ and $\delta_1$ which sends the single object to $1$. So we have two different functors $\delta^{C}_0: C \to [1]\times C$ and $\delta^{C}_1: C\to [1]\times C$ and hence two different functors $F = H \circ \delta^C_0$ and $G = H\circ \delta^C_{1}$. For a morphism $(\sigma,\phi):(i,a) \to (j,b)$ in $[1]\times C$, we get a morphism $H(\sigma,\phi):H(i,a) \to H(j,b)$. For a morphism $f: x \to y$, $F(f) = H(id_0,f) :H(0,x) \to H(0,y)$ and $G(f) = H(id_1,f) :H(1,x) \to H(1,y)$. Lets call $\tau$ the non isomorphism $0\to 1$ in $[1]$, and $H(\tau,id_x) = \kappa_x$ for any $x \in C$. We have the following commutative diagram in $[1]\times C$ : $$ \require{AMScd} \begin{CD} (0,x)@>(id_0, f) >>(0,y)\\ @V(\tau,id_x )VV @V(\tau,id_y)VV \\ (1,x)@>(id_1,f)>> (1,y). \end{CD} $$ This yields the following commutative diagram in $D$ : $$ \require{AMScd} \begin{CD} F(x)@>F(f) >>F(y)\\ @V\kappa_x VV @V\kappa_yVV \\ G(x)@>G(f)>> G(y). \end{CD} $$ which gives back the formula you gave. I will let you convince yourself that given $F,G$ and $\kappa$ which satisfies your formula you can indeed define a functor $[1]\times C\to D$.
So given the functor $H$, $N(H) : N([1]) \times N(C) \to N(D)$ is a morphism of simplicial sets ($N([1]) \times N(C) \simeq N([1]\times C)$ since $N$ commutes with small limits.).
Another important functor is geometric realisation of simplicial sets $|-| : \mathbf{sSet} \to \mathbf{Top}$. This functor commutes with small products and $|N([1])|\simeq [0,1]$ the closed interval. So given $H$, we get a map $$ |N(H)| : [0,1] \times |N(C)| \to |N(D)| $$ in the category of topological spaces, i.e. a homotopy (which can be reversed in this category, so there is no need for invertibility of the $\kappa$'s). Sometimes $|N(C)|$ is shorthandened to $|C|$. You can verify then that $|N(H)|(0,-) = |F|$ and similarly $|N(H)|(0,-) = |G|$.
The other way around is more subtle. Given a topological space $X$ (which is compactly generated and weakly hausdorff let's say) we have $\mathsf{Sing}_{\bullet}(X)$ the simplicial set $[n] \mapsto \mathsf{Sing}_n(X) = Hom_{\mathbf{Top}}(|\Delta^n|,X)$ where for $n\in \mathbb{N}$,$|\Delta^n|$ is the topological space $\{(x_0,\cdots,x_n)\in \mathbb{R}_{\ge 0}^{n+1}, \sum_{i = 0}^{n} x_i = 1\}$. Note that $|\Delta^0| = \{1\}$ and $|\Delta^1| \simeq [0,1]$. So basicaly, $\mathsf{Sing}_0(X)$ is the set of points of $X$, and $\mathsf{Sing}_{1}(X)$ is the set of paths. You can compose paths, but associativity is only up to homotopy and so on, and identities are given by paths that are homotopic to constant paths. But the assignment $X \mapsto \mathsf{Sing}_\bullet(X)$ is functorial by the way I defined and so we get $$ \mathsf{Sing}_\bullet : \mathbf{Top} \to \mathbf{sSet}. $$
Never the less, you can show that for any topological space $X$, $\mathsf{Sing}_\bullet(X)$ is what is called a Kan complex, also called a $\infty$-groupoid (groupoids can be seen as particular kinds of $\infty$-groupoids). Given a homotopy $H:[0,1]\times X \to Y$ and naming $H(0,-) = F$ and $H(1,-)= G$ you get a natural transformation between the $\infty$-functors $\mathsf{Sing_\bullet}(F)$ and $\mathsf{Sing_\bullet}(G)$ from $\mathsf{Sing_\bullet}(X)\to \mathsf{Sing}_{\bullet}(Y)$. Of course as natural transformations between functors between groupoids are invertible, so are the natural transformations of $\infty$-groupoids in some convoluted sense.
Also a particularly neat statement is that the geometric realisation $|\mathsf{Sing}_{\bullet}(X)|$ gives you back the homotopical information of $X$.
The geometric ralisation of a category $C$, since it is a space and so an $\infty$-groupoidy thing, you are loosing information about $C$, basically you are inverting every morphism and that is why you get a homotopy which can be inverted.
So basically to get the reverse picture you have to leave the world of groupoids and categories and explore the world of $\infty$-groupoids and $\infty$-categories.
I will just add to finish that $\mathbf{sSet}$ is a wonderful place to do homotopy theory and category theory, and that combining this two features together we get the theory of $\infty$-categories that Joyal, Lurie and many others developped.