A question about Standard Exponential Family

Question

If $P$ is a regular $\text{SEF}_k$ (standard exponential family) with natural observation $X=(X_1,X_2,\ldots,X_k)^T$, natural parameter $\theta$ and cumulant generating function $\varphi$, then we have

$$ \Sigma = \operatorname{Cov}_\theta[X] = \nabla\nabla\varphi(\theta) = \left(\frac{\partial^2}{\partial\theta_i \, \partial\theta_j} \varphi(\theta) \right)_{i,j=1,\,\ldots\,,\,k} $$

How can we show that?

Answer 1

You have that $A(\eta) = log(Z(\eta)) \rightarrow \frac{\partial}{\partial \eta} log(Z(\eta)) = \frac{1}{z}\frac{\partial}{\partial \eta}\int_{X}f'(x|\eta)dx$.

Where $f'(x)$ is the unnormalized density. Leaving the derivative outside and moving the normalizing constant in:

\begin{equation} \begin{split} \frac{1}{Z}\frac{\partial}{\partial \eta}\int_{X}f'(x|\eta)dx & = \frac{1}{Z}\frac{\partial}{\partial \eta}\int_{X}f'(x|\eta)dx\\ & = \int_{X}\frac{\partial}{\partial \eta}\frac{f'(x|\eta)}{Z}dx\\ & = \int_{X}\frac{\partial}{\partial \eta}\frac{h(x)e^{\eta T(x)}}{Z}dx\\ & = \int_{X}T(x)\frac{h(x)e^{\eta T(x)}}{Z}dx\\ & = \int_{X}T(x)f(x|\eta)dx\\ & = E[T(x)] \end{split} \end{equation}

You can see then that if you took another derivative you would by pulling out another $T(x)$ to form the cross product of $T(x)$ with itself.

Starting at line 4:

\begin{equation} \begin{split} \frac{\partial}{\partial \eta} \int_{X}T(x)\frac{h(x)e^{\eta T(x)}}{Z}dx & = \int_{X}T(x)\frac{\partial}{\partial \eta} \frac{h(x)e^{\eta T(x)}}{Z}dx\\ & = \int_{X}T(x)T(x)^Tf(x|\eta)dx\\ & = Cov[T(x)] \end{split} \end{equation}