Suppose we are doing multiple linear regression:
$\boldsymbol{Y} = X\boldsymbol{\beta}+\epsilon$
and
$\epsilon \sim N(0,\bar{\Sigma})$
Now, suppose that one of the coefficients is defined using some function, for example
$\beta_1 = ax+bx^2$
where $x$ is some quantity of interest, for example a x-coordinate or something.
Now we can estimate $a,b$ and any other coefficients by maximising the log-likelihood of the normal distribution with respect to these parameters. Using these estimates for $a$ and $b$ we can get estimates for $\beta_1$.
We could also find the covariance matrix between $a,b$ and all other coefficients except $\beta_1$ using the Information matrix. Let this covariance matrix be denoted $\Sigma$.
Now my question is how to we obtain the covariance matrix between all the beta coefficients?
If the transformation of $\beta_1$ is linear in $a$ and $b$ (like above) then we can find the desired covariance matrix using
$A\Sigma A^T$
where A is a matrix describing the linear transformation. However what do we do if the transformation is non-linear in $a$ and $b$?
I know the delta method exists and I can obtain the variance in the $\beta_1$ but how can this be extended to find the covariance between say $\beta_1$ and $\beta_2$?
Example:
Suppose instead of estimating $(\beta_0 \beta_1 \beta_2...,\beta_p)$ we want to estimate
$(\beta_0 , \beta_1=exp(ax^2) , \beta_2...,\beta_p)$
where data has been generated using $\beta_1=exp(ax^2)$. Using the log-likelihood we can estimate the most likely values of
$(\beta_0 , a , \beta_2...,\beta_p)$
and we can use the Information matrix to find the covariance of these parameters. My question is how to find the covariance matrix of
$(\beta_0 , \beta_1 , \beta_2...,\beta_p)$.
If $\beta_1$ was linear in a, we could use some transformation of the normal random variable by multiplying by a matrix A to get the covariance matrix $A\Sigma A^T$, but what now when $\beta_1$ is not linear in a?
A general delta method should get you the whole variance-covariance of $\hat{\beta}$, or if you're using MLE you could use the standard score function approach. For example, you can look up how the standard errors of a logistic or a probit regression are calculated, which are both canonical examples of equations that are non-linear in $\beta$.
More specifically, I'm not sure I understand your restriction on $\beta_1$. Could you edit your question to give a more concrete or clear example?