Quantitative approximation by shallow ReLu networks

41 Views Asked by At

Let $f: I \to \mathbb R$ be an $L$-Lipschitz function with compact support. Show that for all $\epsilon > 0$ there is a function $\phi$ given by $$ \phi(x) = \sum_{i=1}^W c_i \mathrm{ReLu}(a_i x + b_i) \quad a_i, b_i, c_i \in \mathbb R$$ where $W \leq 3 \lceil L |I|/\epsilon \rceil$ such that $$ ||\phi - f||_{\infty} \leq \epsilon.$$

Since $\phi$ is a neural network with $1$ hidden layer, width $W$ and ReLu activation function, the general statement follows from some approximation theorem. But I haven't seen any constructive proofs of the sort.

Maybe someone can help with that. I'd also be thankful for any useful references like lecture notes or books that cover this subject (approximation properties of neural networks) in width and depth. In most books about machine learning, this topic is treated rather cursorily or not at all.