Integrating a function around a segment with a finite incontinuity

Question

It always seemed obvious for me that when integrating a function over an infinitesimal segment that contains a finite "jump", we receive $0$. I wonder how that can be justified mathematically. For the sake of the example, say we have $$f=\begin{cases} f_{0} & x\in\left[-a,a\right]\\ 0 & \text{elsewhere } \end{cases}$$ Of course $f_0\neq0$ because otherwise it isn't interesting. How does one show rigourously $$\int_{a-\epsilon}^{a+\epsilon}fdx$$ vanishes? I know the integral is invariant under changes of a finite number of values of $f$, but can that be applied here?