According to this wikipedia page:
https://en.wikipedia.org/wiki/Activation_function
the derivative of the Rectified linear unit (ReLU) function:
$$f(x) = 0 \text{ if }x<0; x \text{ otherwise }$$
has a value of $f'(0)=1$. This surprise me, because on this point I expected undefined derivate or something similar to delta Dirac (compare with the "binary step" function of same page).
Could someone explain why derivative is 1 and not undefined ?
On
Mathematically, you are right; that derivative is undefined. However, in practice, when coding, you need to give it a value so that the program doesn't crash if it should happen to stumble upon $f'(0)$ during calculations. If you have to choose a number, then either $0$ or $1$ would be most natural to pick (since those are the left and right derivatives at $0$), and so, either by random chance or because it actually works better, $f'(0) = 1$ has been picked as a convention.
For $$f(x) = 0 \text{ if }x<0; x \text{ otherwise }$$
at $x=0$ we have the right derivative $ f'_+(0) = 1$ and the left derivative $ f'_-(0) = 0$
Thus $f'(0)$ does not exist.