If Neural Networks are lowsy interpolators then why do we use them alot? Are there better alternatives?

Question

So Neural Nets (NN) are universal function approximators and currently being exploited a lot in various applications to learn underlying function that explains some behaviors, so my question is ,this post claims that neural nets are lousy interpolators and they are not as fast as if we directly program the generic function they aim to approximate, then why are they being used abundantly? Are there better alternatives to neural nets as superior universal approximators or can we work out the actual analytical form of the behaviour that we no longer need NN's?

Answer 1

We use neural networks for tasks where we don't know what to do, and can get a neural network to do it by saying "here, look at these examples, and do that, but more generally". You are exactly right that if we can work out a formula for the behavior we want, then we'd usually do better by implementing the formula than by using a neural network.

The tradeoff is:

• If there is a fast, exact method for solving the problem (such as in the examples you give in the other question, where we want to find $\sin x$), then there's basically no chance for neural networks to be useful.
• If there is an exact but slow method, then it's possible that you could train a neural network to give faster, approximate answers.
• However, if there are known fast approximation algorithms, then they'll usually be a better choice. In particular, we can often prove that an approximation algorithm will have some guaranteed performance - and it is very hard to make guarantees about the output of a neural network.

The most interesting applications of neural networks are to things like image recognition where we expect that there's some nice underlying pattern to what cats look like (or whatever), so there's some hope that the neural network will learn to approximate that pattern, but we don't have any mathematical description of it.