A VAE can generate a latent manifold that is continuous (Manifold Hypothesis).
There are examples of discrete latent manifold, but the problem with such architectures is that the networks are not trainable because discrete functions are not differentiable. Hence different tricks are applied to make them differentiable, such as applying the gradients on the step before the discretatization and essentially skipping the discrete latent space (Straight Through trick, abstract).
My questions:
How does the topology of latent manifold interfere with the training of a deep neural network?
In extension to the above question, Does the manifold must be differentiable at every point to train a neural network (e.g. continuous)?
Is there a formalism that either proves or disproves the above?
Note: I gave an example of a case where through a trick we don't need a differentiable manifold. We transform a continuous manifold to a discrete, however we still apply the gradients to the continuous case (pre-transformation) in order to train the network
Thinking more of my own question here is what I have come up with:
In order for a network to be trainable every layer must have a continuous output, as a consequence the latent space will have to be continuous and any transformation would not have the gradients directly flow through.
The interference on the topology of the manifold would be the same as the topology of the Loss function. i.e. convex functions make for better learning objectives. A convex manifold would in consequence lead to "better" representations.