In the Taylor series expansion of the delta method, could someone help me why it leads to this equation? I understand the first and second components, but the little-o function on the very right is confusing to me. How is this part of the taylor expansion? Is this function somehow equal to the rest of the expansion (remainder of the taylor formula)? Otherwise I have no idea how to make of the little-o function or understand how this can be part of the taylor expansion.
I would appreciate it if someone could explain where the little-o function came from and how the taylor expansion of f(Xn) at theta leads to the existence of this little-o function, thanks.
Typically, the Delta methods is proven using the Mean value theorem. Write $$ \tau_n(f(X_n)-f(\mu))=f'(\mu_n)\tau_n(X_n-\mu), $$ where $\tau_n$ is a normalizing sequence (e.g., $\sqrt{n}$) and $\mu_n$ is a random variable satisfying $|\mu_n-\mu|\le |X_n-\mu|$. When $X_n$ converges in prob. to $\mu$ (i.e., $X_n-\mu=o_p(1)$), $\tau_n(X_n-\mu)=O_p(1)$, and $f'$ is continuous in the nbrhd of $\mu$, \begin{align} \tau_n(f(X_n)-f(\mu))&=f'(\mu)\tau_n(X_n-\mu)+(f'(\mu)-f'(\mu_n))\tau_n(X_n-\mu) \\ &=f'(\mu)\tau_n(X_n-\mu)+o_p(1)O_p(1), \end{align} and $o_p(1)O_p(1)=o_p(1)$.