I'm a bit confused with the expectation notation and how people use it. This one, in particular, is about entropy.
They define Information as
$$I(P) = -log(P)$$
and the entropy as the expectation of Information
$$H(P) = -E_{x\sim P}[log(P)]$$
This is where my confusion comes in. I thought Expectation expanded to an average, ie:
$$-E_{x\sim P}[log(P)] = \frac{1}{N} \sum_{i=1}^{n}log(p_i)$$
but a separate blog expands it out like:
$$-E_{x\sim P}[log(P)] = \sum_{i=1}^{n}log(p_i)$$
Q1: Does expectation expand to a sum or an average? (or is the 1/n simply a constant which can be ignored?
Q2: Is that expansion what the E is short-hand for? ie: empirical average of whatever is in the brackets?
$$E[something] = \frac{1}{N} \sum_{i=1}^{n}something$$
Q3: and does the subscript of expectation, map to whatever the sum indexes?
$$E_{x\sim P}[P + log(Q)] = log(Q) + \frac{1}{N} \sum_{i=1}^{n}p_i$$
You are confusing two things. First, the expected value of a (discrete) random variable $X$ with probability function $P_X(x) =P(X=x)$ is defined as $$E(X)=\sum_x x P_X(x)$$ Notice that the sum runs over all the support of $P$ (i.e., possible values of the random variable).
More in general, $E(g(X))= \sum_x g(x) P_X(x)$ where $g()$ is an arbitrary function.
Second, the sample average, for $n$ realizations of a random variable $X_1, X_2 \cdots X_n$ , is $$\overline X = \frac{1}{n}\sum_{i=1}^n X_i$$
These are very different things. For one thing, the second is a random variable, the first is not.
Now, under certain conditions, for large enough $n$, $\overline X$ converges (in some sense) to $E[X]$. For details, see the law of large numbers.
But the entropy is an expected value: $H(X)=E [ I(X)] = E [ -\log(p(X))]= \sum_x - \log(p(x)) p(x)$