Calculating method of moments estimators for exponential random variables

Question

I'm trying to find the method of moment estimators for $\sigma$ and $\tau$. I have the i.i.d. exponential random variables $X_1, \dots, X_n$ with the density functions

$$f(x; \sigma, \tau)= \begin{cases} \dfrac{1}{\sigma} e^{-(x - \tau)/\sigma} &\text{if}\, x\geq \tau\\ 0 &\text{otherwise} \end{cases}$$

I know that $E(X)=\sigma + \tau$ and $E(X^2)=2\sigma^2+2\sigma \tau + \tau^2$. Then, for method of moments, we set $\frac{1}{n}\sum_{i=1}^{n}X_i=\hat{\sigma}+\hat{\tau}$ and $\frac{1}{n}\sum_{i=1}^nX_i^2=2\hat{\sigma}^2+2\hat{\sigma}\hat{\tau}+\hat{\tau}^2$. But I don't understand how to proceed from here to get the estimators for $\sigma$ and $\tau$, and I don't really understand how $E(X)=\sigma + \tau$ and $E(X^2)=2\sigma^2+2\sigma \tau + \tau^2$ were calculated. For $E(X)=\sigma + \tau$, I tried

$$E(X) = \int_\tau^\infty \dfrac{1}{\sigma} e^{-(x - \tau)/\sigma} \ dx,$$

but this results in $1$, so I don't think it's correct.

So how do proceed from here to get the estimators for $\sigma$ and $\tau$? And how were $E(X)$ and $E(X^2)$ calculated?

Answer 1

$P(\tau \leq x < \infty ) = \int_{\tau}^{\infty} f(x) dx = 1$

$E[X] = \int_{\tau}^{\infty} xf(x) dx = \sigma + \tau$, because this is simply an exponential distribution that has been shifted.

$E[X+c] = E[X] + E[c] = E[X] + c$

Variance $V[X+c] = V[X]$ because adding a constant everywhere does not change the shape of the distribution at all.

$E[X^2] = V[X] + (E[X])^2 = \sigma^2 + (\sigma + \tau)^2$

In your problem, you have two equations in two unknowns, $\hat \sigma$ and $ \hat \tau$. Solve for one (usually in the simpler equation) and substitute into the other.

In writing it all out, I'm going to use some new symbols to save on typing: let $$s = \hat \sigma, t = \hat \tau, y = \frac{1}{n}\sum X_i, z =\frac{1}{n}\sum X_i^2 $$ so our equations are $$y = s+t, z = 2s^2 + 2st + t^2$$ and the goal is to solve for $s$ and $t$ in terms of $y$ and $z$. $$ s = y-t$$ $$ z = 2(y-t)^2 + 2(y-t)t + t^2$$ $$ z = 2y^2 - 4yt + 2t^2 + 2yt - 2t^2 + t^2$$ $$ z = t^2 - 2yt + 2y^2$$ $$ 0 = t^2 - 2yt + 2y^2-z$$ so $$t = \frac{2y \pm \sqrt{4y^2 - 4\cdot1\cdot(2y^2 - z)} }{2}$$ $$t = y \pm \sqrt{y^2 - (2y^2 - z)}$$ $$t=y\pm\sqrt{z-y^2}$$ $$s = \sqrt{z-y^2}$$ so switching notation back, we get $$\hat \sigma = \sqrt{\frac{1}{n}\sum X_i^2 - (\frac{1}{n}\sum X_i)^2}$$ $$\hat \tau = \frac{1}{n}\sum X_i \pm \sqrt{\frac{1}{n}\sum X_i^2 - (\frac{1}{n}\sum X_i)^2}$$ and as the other answer explained, this rearranges in a few more lines of algebra to the answer you gave.

Answer 2

Since $X$ is a shifted exponential with location $\tau$ and scale $\sigma$, it follows that $X - \tau$ is exponential with scale $\sigma$, hence $\operatorname{E}[X - \tau] = \sigma$. This implies $$\operatorname{E}[X] = \sigma + \tau.$$ Moreover, the variance of $X$ is unchanged by a location shift; i.e., $\operatorname{Var}[X] = \sigma^2$, so $$\operatorname{E}[X^2] = \operatorname{Var}[X] + \operatorname{E}[X]^2 = \sigma^2 + (\sigma + \tau)^2 = 2\sigma^2 + 2 \sigma \tau + \tau^2.$$ Of course, we can compute these via integration: $$\operatorname{E}[X] = \int_{x=\tau}^\infty x \frac{1}{\sigma} e^{-(x - \tau)/\sigma} \, dx, \quad \operatorname{E}[X^2] = \int_{x=\tau}^\infty x^2 \frac{1}{\sigma} e^{-(x - \tau)/\sigma} \, dx;$$ however, knowing the properties of the exponential distribution is preferable.

As for the method of moments estimation, we match on the first two raw moments to obtain the system

$$\begin{align} \bar X &= \operatorname{E}[X] = \sigma + \tau, \\ \bar X_2 &= \operatorname{E}[X^2] = \sigma^2 + (\sigma + \tau)^2, \end{align}$$ where $\bar X_2 = \frac{1}{n} \sum_{i=1}^n X_i^2$ is the second raw sample moment. Note that the way the second equation is written allows us to perform a substitution: $$\sigma^2 = \bar X_2 - \bar X^2.$$ Therefore, $$\sigma = (\bar X_2 - \bar X^2)^{1/2}$$ and $$\tau = \bar X - \sigma = \bar X - (\bar X_2 - \bar X^2)^{1/2}.$$ So the desired method of moments estimators are $$\hat \tau = \bar X - (\bar X_2 - \bar X^2)^{1/2}, \\ \hat \sigma = (\bar X_2 - \bar X^2)^{1/2}.$$

Why does this look different? Well, notice $$\begin{align} \bar X_2 &= \frac{1}{n} \sum_{i=1}^n X_i^2 \\ &= \frac{1}{n} \sum_{i=1}^n (X_i - \bar X + \bar X)^2 \\ &= \frac{1}{n} \sum_{i=1}^n \left( (X_i - \bar X)^2 + 2(X_i - \bar X)\bar X + \bar X^2 \right) \\ &= \frac{1}{n} \sum_{i=1}^n (X_i - \bar X)^2 + \frac{2 \bar X}{n} \sum_{i=1}^n (X_i - \bar X) + \frac{1}{n} (n \bar X^2) \\ &= \frac{1}{n} \sum_{i=1}^n (X_i - \bar X)^2 + 0 + \bar X^2, \end{align}$$

where in the last step we have used the fact that $\sum_{i=1}^n X_i = n \bar X$, thus the second term is zero. Hence

$$\bar X_2 - \bar X^2 = \frac{1}{n} \sum_{i=1}^n (X_i - \bar X)^2,$$ and this leads to the alternative formula you referred to. Note that in the linked question, $\theta = 1/\sigma$; i.e., $\theta$ is a rate parameter, not scale.