Casting an expectation as an integral

39 Views Asked by At

I probably picked the most ambiguous title possible for the question I am about to ask. Sorry for that.

I have two random variables, $X$ and $Y$. I am about to define conditional densities and I am not very knowledgeable on how conditional density fits in the probability theory. To avoid complications I assume $Y$ has a discrete distribution and $X$ has a continuous distribution. So then, the conditional density of $X$ given $Y = y$ can be defined without raising too many eyebrows as long as $P\{Y = y\} > 0$. Let's denote this conditional density by $p_{\theta}(X\mid Y=y)$. Here ${\theta}$ is a parameter governing this conditional density. Let $\theta_0$ denote the true value of this parameter and define the following variable.

$$M(\theta) = E_{\theta_0}\left[\log{p_{\theta}(X\mid Y=y)}\right]$$

$E_{\theta_0}$ means that I am taking the expectation under the true distribution of $X$. Here is what I want to prove:

$$M(\theta_0) - M(\theta) = E_{\theta_0}\left[\int\log{\left(\frac{{p_{\theta_0}(x\mid Y=y)}}{{p_{\theta}(x\mid Y=y)}}\right)}p_{\theta_0}(x\mid Y=y)\,d\mu(x)\right]$$

$\mu$ is the Lebesgue measure above.

Unfortunately I don't have much to show as effort as the appearance of an integral in the expression above seems cryptic to me (note that expectation is still there).