I was reading the research paper TOWARDS PRINCIPLED METHODS FOR TRAINING GENERATIVE ADVERSARIAL NETWORKS. I got stuck at Section 2 Lemma 1 and it's proof in Appendix A. I have following questions,
The Lemma $1$ says, $g(Z)$ will have a measure $0$ in $X$ and that too when dimensionality of $Z$ will be less than $X$. What will happen if dimensionality of $Z$ is greater than $X$, may be, which is the case with traditional Artificial Neural Networks.
Is it possible for a set to be continuous and have measure $0$? I don't think so, but I am not very sure. Any opinions?
Following up on $2$, are they proving that Pg won't be continuous since, $g(Z)$ is discrete because it is having measure 0 in X under the condition that dim Z < dim X and dim $g(Z) \le$ dim Z. I read, this document for understanding the essence of 0 measure but couldn't get much for finding the answers to above questions.
Thank you.
P.S. - It would be great if you can explain in layman terms as I am not very involved in this kind of rigorous mathematics. Though, I would love to learn so, it would be fine if you can explain it through some easy examples.
"Pg won't be continuous since, g(Z) is discrete because it is having measure 0 in X under the condition that dim Z < dim X and dim g(Z) <= dim Z"
Like most papers on GAN analysis and convergence, the paper you cite (see top of p. 8) relies on Goodfellow et al's 2014 GAN paper Proposition 1. It has recently come to light that this Proposition is false in the case dim Z < dim X. The reasons are quite simple to understand and are given, along with some simple low dimensional examples, in section 2 of the paper: https://www.researchgate.net/publication/356815736_Convergence_and_Optimality_Analysis_of_Low-Dimensional_Generative_Adversarial_Networks_using_Error_Function_Integrals
You don't need to understand measure theory to understand why. Basically, when dim(z) < dim(x), the generator output PDF is non-unique and degenerate, i.e. contains delta functions. It is the presence of the latter that invalidate the variational calculus argument in the proof of Proposition 1 of the 2014 GAN paper, which relies on continuously differentiable integrands. So the problem is not whether the generator output $\hat{x}=G(z)$ is measurable, but whether the corresponding PDF (confusingly written $p_g(x)$ since $x$ is used as the data variable) is continuous.
I don't have an answer to your question 1 in the context of Arjovsky & Bottou (2017), but the original GAN paper Proposition 1 is valid when dim(z) $\geq$ dim(x). The interesting part is that dim(z) < dim(x) almost always applies for practical GANs (apart from a few toy examples on 2D data sets).