Interpreting a (function of) probability distribution as a random variable?

122 Views Asked by At

Consider, for example, Wikipedia's definition of the entropy of of a discrete random variable $X$:

$$H(X) = \mathbb{E}[I(X)] = \mathbb{E}[-ln(P(X))]$$ Here $\mathbb{E}$ is the expected value operator, and $I$ is the information content of $X$. $I(X)$ is itself a random variable.

Since $I(X)$ is a function of $P(X)$, the PMF $P(X)$ itself is considered to be a random variable that takes on the value $P(x)$ in the event $X=x$. Is my interpretation correct?

A bit of a background: I see this kind of thing a lot in statistics problems dealing with KL divergence (e.g. in EM or variational inference where we often deal with $\mathbb{E}_{q(z)}[logp(z|\theta)]$), and I've always treated the expected value operator like a "macro" where I substitute $\mathbb{E}_{q(z)}[\cdot]$ with $\sum_z \cdot q(z)$ or $\int \cdot q(z) dz$ provided that the argument is some function of some random variable. I'm not sure if this treatment is correct, considering the formal definition of the "expected value operator".

1

There are 1 best solutions below

0
On BEST ANSWER

Yes, the PMF/PDF is a function $P(\cdot)$ that takes in some $x$ and gives you a real umber. In these quantities that you mention, we plug in the random variable itself as the argument, so $P(X)$ is random.

Yes your interpretation of expectation is fine. Note that these quantities are just special cases of functions of random variables. For a random variable $X$ I am sure you understand what $f(X)$ and $\mathbb{E}[f(X)]$ denote. Here we are just thinking of special cases of $f$, such as a PMF/PDF like $P(\cdot)$ or a log likelihood $\log p(\cdot \mid \theta)$