Making Sense of the Exponential Distribution and the Probability Density Function

Question

I read that, due to the memoryless property of exponential distributions, the distribution should be used when the rate of an event is constant during the entire period of time. An example would be the rate of failure for transistors over a number of hours.

But wouldn't this constant rate of an event occurring over time result in a probability density function (PDF) that is a horizontal line? And as such, isn't this incompatible with the exponential PDF desired for exponential distributions?

I'm trying to look at the graphs for exponential distributions (and, thus, their PDF) and reconcile this with the theory I'm reading.

I would greatly appreciate it if people could please take the time to clarify this.

Answer 1

"Memoryless" does not mean the probability of a transistor failure between six weeks from now and six weeks plus one minute from now is the same as the probability of a failure within the next minute. That is what would correspond to a constant density.

Rather, memorylessness means that the conditional probability of a failure between six weeks and six weeks plus one minute, given that the component survives for six weeks, is the same as the probability of a failure within the next minute.

(I am suspicious of the use of this to model failure of transistors. Or light bulbs. One might think transistors age. But time between arrivals of phone calls at a busy switchboard seems plausible. The fact that a phone call came in a minute ago does not make it more likely, nor less, that a phone call will come in within the next five minutes.)

Answer 2

I think the misunderstanding is due to the fact that the term "rate" is often used with different meanings. Sometimes the term "rate" and "probability" are confused, but for me, the definition of rate is the one common in survival analysis and it is an instantaneous measure of change. The instantaneous rate $\lambda(t)$ of a continuous random variable $T$ is defined as. $$\lambda(t) = \lim_{\Delta_t \downarrow 0} \frac{1}{\Delta_t} \mathbb{P}(T\in[t,t+\Delta_t)\mid T \ge t) = \dfrac{f(t)}{S(t)},$$ where $f(t)$ is the density function and $S(t) = \mathbb{P}(T \ge t)$ is the survival function. Therefore in the exponential case, you get a constant rate: $$\lambda(t) = \frac{\lambda e^{-\lambda t}}{e^{-\lambda t}} = \lambda.$$

Answer 3

The best way to understand what is going on with the memoryless property is to go back to the discrete case and think about something like coin tossing or dice rolling.

For example, suppose you have a fair coin. You toss it repeatedly, say once per second, and observe the outcome of each trial until you observe the first instance of heads, at which point you stop. The number of times that you need to toss the coin until you stop, is a geometric random variable--since trials occur at a rate of 1 per second, you can also think of this as the amount of time until the first head is observed.

Now, because the outcome of individual tosses are independent--after all, the coin does not have a way to "remember" what it did before--this distribution has the memoryless property: it is a simple exercise to show that, as long as you have not yet stopped, each time you toss the coin is like the first time you toss the coin in the sense that the previous outcomes have no bearing on the probability distribution of the number of additional tosses you need until you can stop.

As you can see, this is a perfectly natural and intuitive idea--the notion that there do exist random processes that behave in such a way. Students should not have difficulty accepting this.

How long on average do you have to wait to see the first head when the coin is fair? Well, if the outcome is heads on the first trial (which occurs with probability $p = 1/2$), then you waited $1$ second. If the outcome is tails on the first trial (again with probability $1/2$), you have now just spent $1$ second, but the memoryless property means that the additional amount of time to wait to stop is the same as it was before you started flipping. Therefore, $$\operatorname{E}[S] = 1 \cdot \frac{1}{2} + \operatorname{E}[S] \cdot \frac{1}{2} = 1 + \operatorname{E}[S]/2,$$ hence $\operatorname{E}[S] = 2$. On average, you must wait $2$ seconds; or put another way, an average rate of $1/2$ heads flipped per second.

Now say you build a robot to do your coin flipping--after all, once a second may be very tiring--or better yet, you decide to simulate this process on a computer and do away entirely with a physical coin. So now your computer is able to generate a million outcomes every second. But you still want the overall rate at which heads are occurring to remain $1/2$ heads per second, otherwise you are just conducting the same experiment a million times faster. What you really want to do is understand what would happen to the probability distribution of the time-to-first-heads event when you keep the average event rate the same, but make the trials occur more frequently.

So, how do you adjust it? Naturally, what you'd do is make each trial less likely to come up heads: originally it was $p = 1/2$. But with a million trials per second, you could make it a million times less likely each time. And in the limit, as you allow the frequency of trials to increase without bound while keeping the event rate (i.e., heads per second) constant, you get a continuous-time version of the geometric distribution, which is the exponential distribution. And it is not hard to see that this distribution retains the memoryless property of its discrete counterpart, because it is still in some sense the same underlying random process.

On a final note, the use of the exponential failure time model for certain random processes may not be justified, but it is often convenient because of the memoryless property, which as we have seen, does in fact imply a constant failure rate. The failure rate is not to be confused with failure probability in a certain time interval. It is, as noted, a measure of the intensity with which failures occur.

I leave as an exercise for the interested reader to formalize the above.