I am trying to
A regression problem where we have n distincts points, $x_{1}, x_{2},\ldots, x_{n} \in \mathbb{R}$. We can treat observations as independent random variables $y_{1}, y_{2},\ldots, y_{n}$ depending on the given formula:
$$ y_i = \theta_{0} + \theta_{1}x_i^2 + \theta_{2}\exp{(x_i)} + \varepsilon_i $$
where $\varepsilon_i$ are distributed as $N(0,\sigma^2)$. Derive the maximum likelihood estimator for $\theta_{0},\theta_{1},\theta_{2}$ .
Maximising $L(y_n | \theta_0, \theta_1, \theta_2)$ is equivalent to minimising $ \sum_{i=1}^n(y_i - \theta_0 - \theta_1 x_i ^ 2 - \theta_2 e^{x_i}) ^ 2 $
I am trying to use ordinary least squares to minimise $ \sum_{i=1}^n(y_i - \theta_0 - \theta_1 x_i ^ 2 - \theta_2 e^{x_i}) ^ 2 $
but really would want to see a step by step solution to reaching.
Kindest regards
Given obesrvations $(x_1, y_1), \ldots, (x_n, y_n)$ and a linear (in terms of parameters) dependence
$$ y_i = \theta_0 + \theta_1x_i^2 + \theta_2e^{x_i} + \varepsilon_i,\quad i = 1, \ldots, n, $$
one can consider a new set of observations $(y_1, z_{1, 1}, z_{2, 1}), \ldots, (y_n, z_{1, n}, z_{2, n})$ and a new linear dependence
$$ y_i = \theta_0 + \theta_1z_{1,i} + \theta_2z_{2,i} + \varepsilon_i,\quad i = 1, \ldots, n, $$
where
$$ z_{1, i} = x_i^2, \quad z_{2, i} = e^{x_i}. $$
Thus, we have a 2-variate linear regression problem:
$$ \mathbf{Y} = \mathbf{Z}\boldsymbol{\theta} + \boldsymbol{\varepsilon}, $$
where
$$ \mathbf{Y} = \begin{pmatrix} y_1 \\ y_2 \\ \ldots \\ y_n \end{pmatrix}, \quad \mathbf{Z} = \begin{pmatrix} 1 & z_{1, 1} & z_{2, 1} \\ 1 & z_{1, 2} & z_{2, 2} \\ \ldots \\ 1 & z_{1, n} & z_{2, n} \end{pmatrix},\quad \boldsymbol{\theta} = \begin{pmatrix} \theta_0 \\ \theta_1 \\ \theta_2 \end{pmatrix}, \boldsymbol{\varepsilon} = \begin{pmatrix} \varepsilon_1 \\ \varepsilon_2 \\ \ldots \\ \varepsilon_n \end{pmatrix}. $$
Minimizing $\sum_{i=1}^n\left(y_i - \theta_0 - \theta_1x_i^2 - \theta_2e^{x_i}\right)^2$ is equivalent to minimizing $\sum_{i=1}^n\left(y_i - \theta_0 - \theta_1z_{1, i} - \theta_2z_{2, i}\right)^2$, which is equivalent to minimizing $\left\lVert\mathbf{Y}-\mathbf{Z}\boldsymbol{\theta}\right\rVert^2$, i.e.
$$ \min_\boldsymbol{\theta} L(\boldsymbol{\theta}) = \min_\boldsymbol{\theta}\left(\mathbf{Y}-\mathbf{Z}\boldsymbol{\theta}\right)^T\left(\mathbf{Y}-\mathbf{Z}\boldsymbol{\theta}\right). $$
The solution to this problem can be found by solving a so-called score equation
$$ \frac{\partial}{\partial\boldsymbol{\theta}}L(\boldsymbol{\theta}) = 0 \Leftrightarrow -2\mathbf{Z}^T\mathbf{Y} + 2\mathbf{Z}^T\mathbf{Z}\boldsymbol{\theta} = 0 \Leftrightarrow \mathbf{Z}^T\mathbf{Z}\boldsymbol{\theta} = \mathbf{Z}^T\mathbf{Y} \Leftrightarrow \boldsymbol{\theta} = \left(\mathbf{Z}^T\mathbf{Z}\right)^{-1}\mathbf{Z}^T\mathbf{Y}. $$