(edited)
I am learning about the concept of Lipschitz constant and function.
I understand you can write:
$$|f(x) - f(y)| \le M |x - y|$$
(where M is the Lipschitz constant) and thus:
$$M \ge \dfrac {|f(x) - f(y)|} {|x - y|}$$
Of course I realise this has similarities with:
$$ f'(x) = \lim_{h \to 0} \frac{|f(x+h)-f(x)| }{h}$$
Could someone please explain what's the actually proper way mathematically to explain or prove that the Lipschitz constant $M$ is actually the same as the first order derivative of the function $f(x)$ (or is it? and if so under which conditions?).
Say $f$ is defined and continuous on an interval $[a,b]$, and is differentiable on the interior $(a,b)$.
If $M$ is a Lipschitz constant, then so is any $N>M$. You might be more interested in that $L:= \sup_{x\in (a,b)} |f'(x)|$ is the optimal Lipschitz constant if it is finite.
By mean value theorem for every $x,y\in [a,b]$, $x < y$, there exists some $c\in (x,y)\subseteq (a,b)$ such that $$ \left | \frac{f(y) - f(x)}{y - x} \right | = |f'(c)| \le L.$$ That is, $L$ is indeed a Lipschitz constant.
Without loss of generality let $L>0$. By the definition of supremum, there is a sequence $(x_n)_{n\in\mathbb N}$ in $(a, b)$ such that $\lim_{n\to\infty} |f'(x_n)| = L$.
For $n\in\mathbb N$, due to differentiability, there exists some $y_n \in (a,b)$ with $$ \left| \frac{f(y_n) - f(x_n)}{y_n - x_n} - f'(x_n) \right | < \frac{1}{n}. $$ Then, by the (reversed) triangle inequality, it follows $$ \left | \frac{f(y_n) - f(x_n)}{y_n - x_n} \right | \ge |f'(x_n)| - \left | \frac{f(y_n) - f(x_n)}{y_n - x_n} - f'(x_n) \right | \to L, $$ for $n\to\infty$.
That is, every Lipschitz constant needs to be at least $L$.