Show property of exponential families

172 Views Asked by At

Show that if $X$ is a random variable with pdf in an exponential family, then $$\mathbb E\left[\sum_{j=1}^k\frac{\partial\phi_j(\theta)}{\partial\theta_m}t_j(X)\right]=-\frac{\partial \log c(\theta)}{\partial \theta_m}$$ for $m=1,...,d$. Hint: Start from the equality $\int_{-\infty}^\infty f(x|\theta)dx=1$ and differentiate both sides.

The pdf of an exponential family according to my notes is $f(x|\theta)=h(x)c(\theta)\exp\left[\phi(\theta)^Tt(x)\right]$. So following the suggestion, we've got $$\begin{split}c(\theta)\int_{-\infty}^\infty h(x)e^{\sum_{j=1}^k\phi_j(\theta)t_j(x)}dx&=1\\ \log c(\theta)+\log\int_{-\infty}^\infty h(x)e^{\sum_{j=1}^k\phi_j(\theta)t_j(x)}dx&=\log 1=0\\ \frac{\partial}{\partial \theta_m}\log \int_{-\infty}^\infty h(x)e^{\sum_{j=1}^k\phi_j(\theta)t_j(x)}dx&=-\frac{\partial \log c(\theta)}{\partial \theta_m}\end{split}$$

Now I wasn't quite sure how to differentiate the left hand side.

1

There are 1 best solutions below

1
On BEST ANSWER

In this computation one typically assumes that you can push the derivative under the integral. Using the chain rule with the logarithm on the outside and then pushing the derivative under the integral, we obtain

$$\frac{\partial}{\partial \theta_m} \log \int_{-\infty}^\infty h(x) e^{\sum_j \phi_j(\theta) t_j(x)} \, dx \, dx = \frac{\int_{-\infty}^\infty h(x) \frac{d}{d\theta_m}( e^{\sum_j \phi_j(\theta) t_j(x)}) \, dx}{\int_{-\infty}^\infty h(u) e^{\sum_j \phi_j(\theta) t_j(u)} \, du}.$$

The denominator is $1/c(\theta)$ (why?), so the right-hand side becomes $$ \int_{-\infty}^\infty c(\theta) h(x) \frac{d}{d\theta_m}( e^{\sum_j \phi_j(\theta) t_j(x)}) \, dx \int_{-\infty}^\infty \left(\sum_j \frac{\partial \phi_j (\theta)}{\partial \theta_m} t_j(x)\right) \underbrace{c(\theta) h(x) e^{\sum_j \phi_j(\theta) t_j(x)}}_{f(x \mid \theta)} \, dx$$ which is the desired quantity.