Exponential Families defined by Radon-Nikodym Theorem

Question

Let $X \in \mathbb R^d$ be a random vector on space $(\Omega, \mathcal F, \mathbb P)$ and its Laplace transform $\varphi(\theta) := \int e^{\theta\cdot X(\omega)}\mathbb P(d\omega)$ exists for a row vector $\theta \in \mathbb R^d$ and hence so does its cumulant transform $\kappa(\theta):=\log\varphi(\theta)$. Then exponential family is defined to have a probability measure $\mathbb P_\theta$ which satisfies:

$$ \frac{d\mathbb P_\theta}{d\mathbb P}(\omega) = \frac{e^{\theta\cdot X(\omega)}}{\varphi(\theta)}. (*) $$

My questions are as follows.

1. The definition $(*)$ seems to use Radon-Nikodym theorem. That is, $\frac{d\mathbb P_\theta}{d\mathbb P}$ is the Radon-Nikodym derivative of $\mathbb P_\theta$ with respect to $\mathbb P$. In other words, $\mathbb P_\theta$ is defined by the following integration. For a measurable set $A$, $$ \mathbb P_\theta(A) := \int_A \frac{e^{\theta\cdot X(\omega)}}{\varphi(\theta)} \mathbb P(d\omega). $$ Is this the correct way to interpret the definition $(*)$, please?
2. If my above interpretation is correct, I want to verify the fact that $\mathbb E_\theta X = \frac{\partial\kappa}{\partial\theta}(\theta)$. $$ \mathbb E_\theta X = \int_\Omega X(\omega) \mathbb P_\theta(d\omega) = \int_\Omega X(\omega) \frac{\mathbb P_\theta(d\omega)}{\mathbb P(d\omega)}\mathbb P(d\omega) = \int_\Omega X(\omega) \frac{e^{\theta\cdot X(\omega)}}{\varphi(\theta)}\mathbb P(d\omega) = \frac{\varphi'(\theta)}{\varphi(\theta)} = \kappa'(\theta). $$ In the third equality above I substituted $\frac{\mathbb P_\theta(d\omega)}{\mathbb P(d\omega)}$ by $\frac{d\mathbb P_\theta}{d\mathbb P}(\omega)$. Is this correct, please? And why? Moreover, I am very uncomfortable with doing things like $\frac{\mathbb P_\theta(d\omega)}{\mathbb P(d\omega)}\mathbb P(d\omega)$. What exactly does this mean, please? How to understand it, please?

Answer 1

I want to clarify some notation. We usually omit $\omega$ in the integation as it would makes things too messy. I will use $dP(\omega)$ or just $dP$ for $P(d\omega)$. The $\omega$ simply means what kind of element of the universe $\Omega$ we are dealing with and we usually omit it for notation convenience.

1. The meaning of $\frac{dP_θ}{dP} (\omega)=e^{θ⋅X(ω)}/φ(θ)$ is just $P_\theta(A)= \int_A e^{θ⋅X(ω)}/φ(θ) dP(\omega)$ as you said. Indeed, this is the usual way in defining the exponential family. And if there is another measurable $g$ s.t. $\int_A gdP=\int_A e^{θ⋅X}/φ(θ) dP,\forall \;A$ measurable, then one can show that $g= e^{θ⋅X}/φ(θ),P-a.s. $. So the definition of exponential family is well-defined. You don't need to use Radon-Nikodym theorem as you already have the density $e^{θ⋅X(ω)}/φ(θ)$. Well, this density is called Radon-Nikodym derivative and denoted as $\frac{dP_θ}{dP}$.

2. It is easy to verify $\int_A f dP_\theta=\int_A f e^{θ⋅X(ω)}/φ(θ) dP$ for any $P-$integrable $f$ by first consider indication function and then linearity of intergal and then MCT and seperate general $f$ to positive and negative part. This means that we could simply substitute $dP_\theta$ by $e^{θ⋅X(ω)}/φ(θ)dP$ or$\frac{dP_\theta}{dP} (\omega) dP$(which makes you uncomfortable ) when computing integral.

In general, $\frac{P_\theta}{P} (d\omega)= \frac{dP_\theta}{dP} (\omega)$ are just some notations and they mean the function(or density) s.t. $P_\theta(A) =\int_A \frac{dP_\theta}{dP} (\omega) dP(\omega),\forall A$ measurable.

I hope this clarify something make you confused.

Answer 2

Yes, the notion $$ \frac{\mathrm{d} \mathbb{P}_\theta}{\mathrm{d}\mathbb{P}}(\omega)=\varphi(\theta)^{-1}\mathrm{e}^{\theta \cdot X(\omega)} $$ means exactly that the measure $\mathbb{P}_\theta$ has density $\varphi(\theta)^{-1}\mathrm{e}^{\theta \cdot X(\omega)}$ with respect to $\mathbb{P}$, that is, $$ \mathbb{P}_\theta(A)=\int_A \varphi(\theta)^{-1}\mathrm{e}^{\theta \cdot X(\omega)}\,\mathbb{P}(\mathrm d\omega),\quad A\in\mathcal{F}. $$ Note that this shows that $$ \int Y(\omega)\,\mathbb{P}_\theta(\mathrm d\omega)=\int Y(\omega)\varphi(\theta)^{-1}\mathrm{e}^{\theta \cdot X(\omega)}\,\mathbb{P}(\mathrm d\omega)\tag{1} $$ holds for all indicator functions $Y$, i.e. $Y=\mathbf{1}_A$ for some $A\in\mathcal{F}$. A standard argument then shows that $(1)$ holds for all measurable and integrable $Y$. In particular, $(1)$ holds for $X$ (if you assume $X$ is integrable) and hence $$ \mathbb{E}_\theta[X]=\int X(\omega)\,\mathbb{P}_\theta(\mathrm d\omega)=\int Y(\omega)\varphi(\theta)^{-1}\mathrm{e}^{\theta \cdot X(\omega)}\,\mathbb{P}(\mathrm d\omega). $$