Is there conjectures in deep learning theory?

Question

I often read that deep learning suffers from a lack of theory, compared to classical machine learning. I mean that deep learning has shown to be a powerful tool in practice but there is no proof of this effect in theory. Which leads to my question: Is there some conjectures in deep learning theory? What should be proven mathematically to build a real deep learning theory?

Answer 1

I don't know of any conjecture that captures all theoretical problems raised by deep learning . Also, there is not much consensus on how to properly formalize the questions. For example Naftali Tishby's group uses an information theoretic approach , while others - like Sanjeev Arora ,looks more to the (non-convex) optimization problems and different simplified or specific architectures (random or linear networks).

A good theory of deep learning should explain two things:

1.Why they generalize so well ,although the standard learning theory indicate that they should overfit .

2.Why the optimization is practically so easy ,although the current theory say it is hard.