Can we also say that convexity implies $f'(x) \geq \frac{f(x) - f(y)}{x-y}$ ($\ \forall x \neq y \in \mathbb{R}$) and vice-versa? (Even if not equivalent statements, maybe at least one sided implication holds?)
If $x > y$ and $f$ is increasing, then $f'(x)(x-y) \geq f(x) - f(y)$ certainly holds (as the derivative at $x$ exceeds the derivative at any of the points in $(x,y)$. Since $\frac{f(x)-f(y)}{x-y}$ is the derivative of some $z \in (x,y)$ which we can show using the mean value theorem, the inequality holds.)
But is that true without these (above) restrictions (like $x>y$ and $f \uparrow$)?
Your inequalities do not hold for all $x,y, x\neq y$.
If $f$ is assumed to be differentiable then it can be shown that $f(y)-f(x)=\int_x^{y} f'(t)dt\leq f'(y) (y-x)$ provided $x<y$.
The equality here is a special result for convex functions and the inequality follows from the fact that $f'$ is increasing.
What we have is $f'(y) \geq \frac{f(x) - f(y)}{x-y}\geq f'(x)$ for $x<y$.