I was wondering what is the effect if I replace the second derivative of the log-likelihood ("Likelihood" hereafter) function with its expectation in a higher-order Taylor expansion of the likelihood function.
Let the observations be i.i.d., $\ell(\theta)$ be the sample mean of individual likelihood over $N$ observations, and l{r}(theta) be the r-th derivative of l(theta) w.r.t. theta. A Taylor expansion of the likelihood around the truth (theta_0, scalar) and evaluated at the estimate (theta_hat, scalar) gives
l(theta_hat) = l(theta_0) + l{1}(theta_0)(theta_hat-theta_0) + 1/2 l{2}(theta_0)(theta_hat-theta_0)^2 + ... + 1/k! l{k}(theta_0)(theta_hat-theta_0)^k + Op[N^(-k-1/2)].
$$ \ell(\hat\theta) = \ell(\theta_0) + \ell'(\theta_0)(\hat\theta-\theta_0) + \frac 1 2 \ell''(\theta_0)(\hat\theta - \theta_0)^2 + \cdots + \frac 1 {k!} \ell^{(k)}(\hat\theta - \theta_0)^k + O_p(N^{-k-1/2}) $$
If k were 2, we could l{2}(theta_0) with E[l{2}(theta_0)] with no problem. Now let k>2 and I want the expansion to be correct in expectation. Can I still replace l{2}(theta_0) with E[l{2}(theta_0)]?
I understand that, typically,
$$ \ell^{(2)}(\theta_0) = E\left[ \ell^{(2)}(\theta_0) \right] + O_p(N^{-1/2}) $$
such that, in some applications, replacing E[l{2}(theta_0)] with l{2}(theta_0) will introduce a nonneglectable bias to the order of Op(N^-1/2). But what is the effect, to what I want, of the replacement that I explained above?
I'm sorry that you are seeing this in such a bad format. I'm new to this place and they don't let me upload pictures....