Find the maximum likelihood estimator of b (Regression coefficient)

Question

Consider the regression model: $y_i = bx_i + e_i,\quad 1 ≤ i ≤ n$,

Suppose that $x_i$’s take values −1 or +1 and $e_i$’s have density $f(t) ={\frac{1}{2}}e^{−|t|}, t \in \mathbb{R}$.

Find the maximum likelihood estimator of $b$.

Therefore $\; y_i-bx_i \sim \epsilon \quad \text{,which follows}\quad f(t) ={\frac{1}{2}}e^{−|t|}\\ \therefore f(y,b,x_i)= {\frac{1}{2}}e^{−|y_i-bx_i|}\\ \Rightarrow L(y,x_i,b) = {\frac{1}{2}}^n e^{−\sum|y_i-bx_i|}\\ \Rightarrow \frac{\partial\log L(y,x_i,b)}{\partial b} = -\frac{\partial{\sum |y_i-bx_i|}}{\partial b} $

Any ideas about how to proceed??

Answer 1

In this case your log-likelihood function is:

$$\ell (b) = - \sum_{i=1}^n | y_i - b x_i |.$$

Since your $x_i$ values are either negative or positive one, we can let $\mathscr{X} \equiv \{ i = 1,...,n | x_i =1 \}$ be the set of indices for which this explanatory variable is one. We then have:

$$\ell (b) = - \sum_{i \in \mathscr{X}} | y_i - b | - \sum_{i \notin \mathscr{X}} | y_i + b |.$$

Taking $\tfrac{d}{dx}|x| = \text{sgn}(x)$ the score function is:

$$\begin{equation} \begin{aligned} \frac{d\ell}{db} (b) &= - \sum_{i \in \mathscr{X}} \text{sgn}( y_i - b ) - \sum_{i \notin \mathscr{X}} \text{sgn}( y_i + b ) \\[6pt] &= \sum_{i \in \mathscr{X}} \text{sgn}( b - y_i ) - \sum_{i \notin \mathscr{X}} \text{sgn}( b + y_i ). \\[6pt] \end{aligned} \end{equation}$$

The score equation is:

$$\sum_{i \in \mathscr{X}} \text{sgn}( \hat{b} - y_i ) = \sum_{i \notin \mathscr{X}} \text{sgn}( \hat{b} + y_i ).$$

The MLE $\hat{b}$ is any value that balances these signs. It is likely that this will be an interval of values, and so in this case you will need to choose a value in the interval arbitrarily (e.g., the midpoint). It should be possible to solve this equation numerically to obtain $\hat{b}$ by checking values over intervals between the appropriate $y_i$ values.

Answer 2

A friend of mine who is not in MSE deserves credit for this answer.

The likelihood function of b is :-$$ L(b,y_i,x_i)\,\,\,=\,{\frac{1}{2}}^n e^{−\sum|y_i-bx_i|}\\ \Rightarrow log \, L(b,y_i,x_i)= c - {\sum |y_i-bx_i|} $$ Now need to maximize the loglikelihood function wrt b, ie minimize ${\sum |y_i-bx_i|}$. Note that $x_i $ can only take values of $\pm 1$.

$\therefore |x_i|=1$, Using this we can write $${\sum |y_i-bx_i|} \Rightarrow ({\sum |y_i-bx_i|})\frac{1}{|x_i|} \Rightarrow ({\sum |\frac{y_i}{x_i}-b|})$$

Here I am going to use an interesting result ${\sum |t_i-b|}$ is minimized if $b=median(t_1,t_2,t_3,....t_n)=t_m$. Using this result , we can say that $${\sum |y_i-bx_i|} \text{ is minimized at }z_m\, \text{where }z_m=median(\frac{y_1}{x_1},\frac{y_2}{x_2},\frac{y_3}{x_3},...\frac{y_n}{x_n})$$

$$\Rightarrow L(b,y_i,x_i)= c - {\sum |y_i-bx_i|} \text {is maximized at }z_m $$

$\therefore$ the MLE of b is $ median(\frac{y_1}{x_1},\frac{y_2}{x_2},\frac{y_3}{x_3},...\frac{y_n}{x_n})$