A physical device can be in three states: A, B, C. The device has an underlying Markov Chain and Markov Process to describe time jumps.

Question

A physical device can be in three states: A, B, C. The device operates as follows (all time units are in hours):

• The device spends an exponentially distributed amount of time in state A (with mean of 12 minutes) and then with probability 0.6 goes to state B, and with prob. 0.4 goes to state C.
• When in state B, the device moves to state C after an Exp(3) amount of time.
• When in state C, the device goes to state A at rate 1/hour, and to state B at rate 2/hour. Let $X_t$ represent the device state at time $t$, and suppose $X_0 = A$. Compute:

(a) Probability the device is in state 'A' after 30 minutes.

(b) Probability the device is in state 'A' after 30 minutes given that it was in state 'B' after 5 minutes and in state 'C' after 10 minutes.

(c) The long-run proportion of time the device spends in state `A'.

So when approaching this problem it is easy for me to figure out the underlying markov chain and its corresponding p transition matrix. I am struggling with approaching how to find the jump rate matrix as well as how to approach part a. I think if I understood how to determine the time portion, the other parts should follow fairly naturally, but I don't get how to use the exponential distribution information given.

Answer 1

The key is to understand how to convert between different descriptions of exponential distributions and Markov chains.

For exponential distributions, the following are equivalent:

• Sleep for an $\operatorname{Exp}(\lambda)$ time before waking up.
• Sleep for an exponentially distributed time with mean $\lambda^{-1}$ before waking up.
• Transition from sleep to waking up at a rate of $\lambda$.

For a Markov chain, there are two ways to view transitions: the "call center" model and the "direct line" model. The following are equivalent:

• Wait for $\operatorname{Exp}(\lambda)$ time for the phone to ring. Then get a call from Alice with probability $p_A$, from Bob with probability $p_B$, from Charlie with probability $p_C$, etc.
• Wait for $\operatorname{Exp}(\lambda p_A)$ time for a phone call from Alice, or for $\operatorname{Exp}(\lambda p_B)$ time for a phone call from Bob, or for $\operatorname{Exp}(\lambda p_C)$ time for a phone call from Charlie, etc. (Whichever comes first.)

In other words, the possible transitions from a state can be thought of as having several competing exponential distributions going at once, and picking the smallest, or as having a single exponential distribution going, and picking which way to go once it's done.

The driving property behind this equivalence is that for exponential distributions, we have $$ \min\{\operatorname{Exp}(\lambda_1), \operatorname{Exp}(\lambda_2), \dots, \operatorname{Exp}(\lambda_n)\} = \operatorname{Exp}(\lambda_1 + \lambda_2 + \dots + \lambda_n). $$