From Convex Optimization:
In the red box below, how does differentiability imply that there exists a $t$ such that this function is negative?
$a \le_K b$ is a partial ordering on a proper cone $K$ that means $b - a \in K$ and $\le_{K^*}$ is the partial ordering on the dual cone.
It's not the function, but rather the derivative, that is negative.
There is the assumption that $f(y) < f(x)$, so at some point on the path from $x$ to $y$, we need to move down, i.e. have negative derivative.