Understanding this math notation in probability?

Question

I don't get this notation at all and cannot find a place to start with understanding this: $$\mathcal L_D=-\mathbb E_{x\sim P}[\log(D(x))]-\mathbb E_{\hat x\sim Q}[\log(1-D(\hat x))]$$ I don't get $\mathbb E$ there. Does it mean expectation? The equation says it's "negative log-likelihood". I understand that $D(x)$ is either 1 or 0. What does the $\mathbb E_{x\sim P}$ mean, especially in the context of statistics?

Assuming that $E$ is expectation, what does it mean to have an expectation of a probability distribution $P$? Does it imply the mean of the distribution, and in that case what does $\hat x$ typically mean? Unfortunately the paper I'm reading doesn't spell these out so I'm guessing I lack a bit of statistics background here.

Answer 1

$\mathbb{E}_{x\sim P}[f(x)]$ is the mean of $f(x)$ if $x$ has distribution $P$.

Answer 2

It means the expected value of the function $\log(D(x))$ where $x$ is drawn from distribution $P$, i.e. $$ \mathbb{E}_{x\sim P}[\log(D(x))] = \int \log(D(x)) \cdot P(x) \cdot dx. $$

Note that if $D(x) = 0$ that $\log(D(x))$ is not defined. Similarly, if $\hat{x} = 1$ then $\log(1 - D(\hat{x}))$ is not defined. So it's more likely that $D(x) \in (0,1)$ rather than $D(x) \in \{0,1\}$. Or at least there are some technical matters to work out here.

The expected value of the probability (or log-probability) of a random variable is a very useful quantity in statistics and information theory. Probability density functions (and probability mass functions) are just functions which take arguments. So you can think of their expectations as just the "average probability score" (or average log-probability score) of an element sampled from from the distribution.

Why is this useful? Well, if the distribution is over very large set of items, each with a small probability of being chosen, then the "typical probability score" of a randomly drawn element will be small. And if instead the distribution is highly concentrated around a small number of elements, then the "typical probability score" of a randomly drawn element will be rather large in comparison. Thus the expected log-probability provides a measure of how much randomness the probability distribution contains, and it's useful in applications such as data compression. See Shannon entropy for more details.

Maximizing the expected log-probability is also useful for problems in statistics where some data is hidden or unobserved. See expectation maximization for more details.