Intuition on fisher information on $n$ observations and its relationship with one observation

910 Views Asked by At

For a sample with a set of iid random variables, $X_1,X_2,\ldots,X_n$ with common parametric family $f(x;\theta)$, its Fisher information is defined to be:
$$I_n(\theta):= -\mathbb{E}\left(\frac{\partial ^2 \ell}{\partial \theta^2}\right)$$ where $\ell$ is the log-likelihood function of the random sample, so $\ell (\theta;x_1,\ldots,x_n) = \ln \left(\prod_{i=1}^{n} f(x_i;\theta)\right)$.

In my notes it says that we can show:
$$I_n(\theta) = n I_1(\theta).$$

What is the intuition behind this Fisher Information relationship? Actually what is a "layman's" explanation of the Fisher Information? Its definition was given by my lecturer, but it wasn't really explained what is really represented.

1

There are 1 best solutions below

0
On

Note that for regular families, you have that $$ I_X(\theta) = Var\left( \frac{\partial}{\partial \theta}\ln f (\theta;X) \right), $$ i.e., Fisher'e information is the variance of the sensitivity of the (log) density to infinitesimal changes in the parameters of interest. I.e., as bigger the variance, namely the density functions changes more abruptly w.r.t $\theta$, then each observation will bear more "information" about the parameter of interest.

And for the second, just observe the linearity of Fisher's information, i.e., let $X_1$ and $X_2$ be two i.i.d random variables, then
$$ I_{X_1, X_2}(\theta) = I_{X_1}(\theta) + I_{X_2}(\theta) = 2I_{X_1}(\theta). $$
Namely, each observation brings the same amount of "information" on $\theta$, thus a sample size of $n$ i.i.d observations gives us $nI_X (\theta)$.