In numerical simulations, one often deals with non-differentability.
For example, a neural network is non-differentiable when its activation function is $\max\{x, 0\}.$ This leads to optimizing a non-differentiable function.
However, in practice often people justifies taking the derivative of these functions by ignoring the point of non-differentiability by claiming that since such points forms a set of measure zero, therefore it does not matter in practice.
Do you agree or disagree with this notion? Is it true that non-differentability is a non-issue in numerical simulations/practice if these points form a set of measure zero?
Unless I am doing integration, then I be would extremely reluctant to ignore a set of measure zero.
For example, there are real functions $f : \mathbb{R} \rightarrow \mathbb{R}$ which are continuous on the irrational numbers $\mathbb{R}\setminus \mathbb{Q}$ and discontinuous on the rational numbers $\mathbb{Q}$. Since $\mathbb{Q}$ is countable it has measure $0$.
I cannot say this with certainty, but I suspect that when points of discontinuity are being ignored to no ill effect, then there only finitely many such points.
I can say with certainty that non-differentiability is an issue in applications which require not only a result, but also a reliable error estimate. Many methods for integrating differential equations and estimating the error require all relevant functions to exhibit a certain degree of differentiablity. In the absence of sufficient smoothness the order of accuracy of your method can be reduced and any such reduction must be detected as it affects the error estimate. Failure to do so will lead to overly optimistic error estimates which are useless. Moreover, higher order methods typically require a higher degree of differentiability than low order methods.
In short, if your problem is not sufficiently smooth then you have to be extra careful when you choose your method and your error estimate.