I have ran into it a few time in my books and it seems to be a function that tends to $0$ as its argument tends to $0$. They are all written in the same notation ($\varepsilon(x)$) and we do not use indices as all the epsilon functions behave the same way. Is it related to O-notation?
I could not find any useful sources.
A example can be seen in the comments
Most likely it is little-$o$ notation. For example, in John M.H. Olmsted - Advanced calculus-Prentice Hall (1961), page 69 you can find definition for differentiation for function of one real variable: $$\Delta y=\frac{dy}{dx}\Delta x + \varepsilon \Delta x$$ where $\varepsilon$ is infinitesimal, as wrote author, i.e. $\varepsilon(\Delta x)\to 0$, when $\Delta x \to 0$.
Farther, in same book, in page 267 for definition for differentiation for multiple variables is used formula: $$\Delta u = \frac{\partial u}{\partial x}\Delta x+\frac{\partial u}{\partial y}\Delta y +\varepsilon_1 \Delta x + \varepsilon_2 \Delta y$$ where again $\varepsilon_1, \varepsilon_2$ are infinitesimals i.e. tending toward zero as $\Delta x$ and $\Delta y$ tend toward zero.
At end let me bring one version of definition little-$o$: $$o(f),x\to x_0=\{g: \exists \varepsilon(x), \lim\limits_{x\to x_0} \varepsilon(x) = 0, \exists U_\delta(x_0), \forall x \in U_\delta(x_0), g(x)=f(x) \varepsilon(x) \}$$ where $U_\delta(x_0)$ is neighbourhood of $x_0$. This definition works as for single, so for multiple, real variables.