Regression maximum likelihood

Question

Given this regression model: $y_{i}=\beta_{0}+\beta_{1}x_{i}+E_{i}$.

All the assumptions are valid except that now: $E_{i}\sim N(0,x_{i}\sigma^{2})$

Find Maximum likelihood parameters for $\beta_0$,$\beta_1$

Answer 1

What makes the model interesting is that the variance is proportional to the data $x_i$. This implies that $x_i\geq 0$. I am going to start by assuming that $x_i>0$. But once we've found the answer in that case I'll discuss the situation when that assumption does not hold; i.e., that $x_i=0$ for at least one $i$.

Let us define the following quantities: $$\bar{y}_i = x_i^{-1/2} y_i, \quad \bar{x}_{i-} = x_i^{-1/2}, \quad \bar{x}_{i+} = x_i^{+1/2}, \quad \bar{E}_i = x_i^{-1/2} E_i$$ Then the equation becomes $$ \bar{y}_i = \beta_0 \bar{x}_{i-} + \beta_1 \bar{x}_{i+} + \bar{E}_i, \quad \bar{E}_i \in \mathcal{N}(0,\sigma^2)$$ What we have now is a standard, unweighted linear regression in two variables.

To solve this we define two matrices $$X = \begin{bmatrix} \bar{x}_{1-} & \bar{x}_{1+} \\ \vdots & \vdots \\ \bar{x}_{n-} & \bar{x}_{n+} \end{bmatrix}, \quad Y = \begin{bmatrix} \bar{y}_1 \\ \vdots \\ \bar{y}_n \end{bmatrix}$$ Then we have $\beta=(\beta_0,\beta_1)=(X^TX)^{-1}X^TY.$ But the special structure of $X$ and $Y$ leads to some interesting looking structure when we multiply it out: \begin{equation} X^TX = \begin{bmatrix} \sum_i x_i^{-1} & n \\ n & \sum_i x_i \end{bmatrix}, \quad X^T Y = \begin{bmatrix} \sum_i x_i^{-1} y_i \\ \sum_i y_i \end{bmatrix} \qquad \beta = \begin{bmatrix} \beta_0 \\ \beta_1 \end{bmatrix} = \frac{1}{ n^2 - \sum_i x_i^{-1} \sum_i x_i} \begin{bmatrix} n \sum_i y_i - (\sum_i x_i) \sum_i x_i^{-1} y_i \\ n \sum_i x_i^{-1} y_i - (\sum_i x_i^{-1}) \sum_i y_i \end{bmatrix} \end{equation}

What happens if we cannot assume $x_i>0$? Well, in that case the original relation reduces to $y_i=\beta_0$ and $E_i=0$. So that means that $\beta_0$ is forced to be equal to $y_i$ in that case. If there are multiple values of $x_i>0$, then we have two scenarios:

• All of the corresponding values of $y_i$ are identical, in which case $\beta_0=y_i$.
• The corresponding values of $y_i$ are not identical, in which case the model is infeasible and cannot be solved.

If the model is feasible, then we fix $\beta_0$ and eliminate those $(x_i,y_i)$ pairs from the model, leaving us with a single-variable regression. I hope you don't mind if I skip right to the result: $$\bar{z}_i = \beta_1 \bar{x}_{i+} + \bar{E}_i, \quad \bar{z}_i \triangleq \bar{y}_i - \beta_0 \bar{x}_{i-}\quad\Longrightarrow\quad \beta_1 = \frac{(\sum_i y_i) - n\beta_0}{\sum_i x_i}.$$