Mathematics (provable properties) of autoencoder neural networks

Question

I'm looking for references about the mathematical theory of autoencoder neural networks. Specifically, provable properties about autoencoders (i.e.: theorems and lemmas).

"Properties" could include claims about convergence, claims about comparison to linear methods (e.g.: PCA), claims about properties in the limit as the autoencoders get deeper, etc. My interest specifically is on the use of autoencoders as a dimensionality reduction technique.

If there is a single reference which collects different aspects of this theory that would be wonderful, but individual papers would be great as well. Limiting discussion to relatively simple architectures that make deriving theory more feasible is perfectly fine!

Is information-bottleneck methods relevant to this discussion? Any references about that would also be wonderful.

More generally, I am also looking for references about the mathematics of non-linear dimensionality reduction techniques. (The emphasis again being on the mathematical theory and provable claims).

I have been having trouble finding references which focus on provable theory, so I'm hoping someone with a bit more experience can guide me through the theory that currently exists.

Thanks!

Answer 1

I don't think this is quite what you're looking for, but Chapter 2 section 5 of Neural Networks and Deep Learning: A Textbook by Charu C. Aggarwal, contains a discussion of autoencoders. It's probably not mathematically rigorous enough for you, but the author also includes references to other resources in his bibliographic notes (Chapter 2 section 9), so at least he provides some further exposure to the literature on autoencoders.

Answer 2

As a starter, how about this reference:

Pierre Baldi and Kurt Hornik. “Neural networks and principal component analysis: Learning from examples without local minima”. In: Neural Networks 2 (1989), pp. 53–58.