Proof of the existence of the latent space in deep learning

Question

I'm working and studying on generative models. I realized that a strict and mathematical background on the latent space is required. Is there any proof of the existence of the latent space for a manifold? Below is what I've understood about the manifold hypothesis and the latent space:

Manifold hypothesis says; the real world data lie on the lower-dimensional manifold embedded in the original high-dimensional space.

and

A latent space is the lower-dimensional representation of the manifold.

That is, the manifold itself is the lower-dimensional object but embedded (or represented) in the high dimension. For example, consider a high-dimensional space $\mathcal{X}\subset\mathbb{R}^N$ and a manifold $\mathcal{M}\subset\mathcal{X}$. Then, there exist the lower-dimensional representation of the manifold $\mathcal{Z}\subset\mathbb{R}^d$, the corresponding non-linear transformation $T:\mathcal{Z}\to\mathcal{M}$, and the inverse transformation $T^{-1}:\mathcal{M}\to\mathcal{Z}$, where $d \ll N$. In the generative model, the generator network $G$ plays a role of the transformation $T$.

However, is it (mathematically) true (proved)? How can I prove the existence of the latent space?

Answer 1

One way to interpret this is: It is your generative model or transformation $T$ that create a manifold $\mathcal P$. If the generated examples covers the target dataset exactly (all the generated samples are good), then this manifold coincide exactly with our desired data manifold $\mathcal M$, $\mathcal P=\mathcal M$, and provides a parametrization of $\mathcal M$.

---

As for proof, I think the equivalent condition of existance of latent space is that the "data manifold" is homeomorphic to $\mathbb R^d$ globally (invertible continuous map exist), which is an empirical question.

I'm not sure about existence proofs of the latent space, but there are proofs of non-existance. For example,

1. if the data manifold $\mathcal M$ is of different dimensions from your latent space $\mathbb R^d$, then there cannot be homeomorphism between them.
2. if the data manifold $\mathcal M$ is of non-trivial topology, for example it has the topology of a sphere $S^d$, then it cannot homeomorphic to latent space $\mathbb R^d$.
3. if the data "manifold" is not a manifold (in mathematical sense) at all, there cannot be homeomorphic to $\mathbb R^d$. This is very possible. Manifold requires the local dimensionality of anywhere on data manifold is $d$, which may not be true.

So in some sense, I may not take the "manifold" in manifold hypothesis as strictly as in differential geometry....

---

BTW, I'm also really interested in the geometric interpretation of generative models and I've worked on these ideas a bit! The Geometry of Deep Generative Image Models and its Applications