What is the difference between observed information and Fisher information?

Question

I have in literature saying the observed information $J(\theta)$ is equal to the Fisher information $I(\theta)$. They are given different donations and same parameter. It is not clear why if equal they have different donations. Could anyone please explain?

Answer 1

Let $X_1,...,X_n \sim f(x;\theta)$. Fisher information is a theoretical measure defined by $$ \mathcal{I}(\theta) = - \mathbb{E}\left[\frac{\partial^2}{\partial\theta^2}\ln f(x:\theta) \right], $$ where $\theta$ is the unknown parameter of interest, hence for sample of size $n$ and MLE $\hat{\theta}_n$, you can estimate the fisher information by $n\mathcal{I}(\hat{\theta}_n)$.

Observed information is defined by $$ \mathcal{I}_{obs}(\theta) = - n\left[\frac{1}{n}\sum_{i=1}^n\frac{\partial^2}{\partial^2 \theta}(\ln f(x_i:\hat{\theta}_n)) \right], $$
which is simply a sample equivalent of the above. So, as you can see, these two notions defined differently, however if you plug-in the MLE in fisher information you get exactly the observed information, $\mathcal{I}_{obs}(\theta)=n\mathcal{I}(\hat{\theta}_n)$.
To show it for a pretty general case, you can work out the algebra for a single parametric exponential family distribution (it is a straightforward calculations).