Tight Rope Walk DTMC

Question

I struggle with finding long-run rates of more advanced problems such as the one below. If somebody could give me a detailed explanation for the answer I would really appreciate it!

Consider an infinite population of people that walk along a tightrope over a chasm 100ft deep from point A to point B. The path consists of positions 0, 1, 2, ..., 4 where position 0 corresponds to point A, and 4 to point B. At any point in time at most one person is on the tightrope, and it takes the person exactly 15 minutes to move from position i to i+1, $0 \leq i \leq 3$. While a person is traveling from i to i+1, he/she falls with probability p, $0<p<1$, independent of everything else. As soon as a person makes it to point B, or a person falls, at the beginning of the next 15-minute interval a new person begins moving along the tight rope from point A. Compute the long-run rate at which people fall (in terms of people per hour) while between positions 2 and 3 on the tight rope. (Note: previously we proved that this process is a DTMC and and that it has a unique limiting distribution).

Answer 1

This Markov chain has 4 states: 0, 1, 2, and 3. At the beginning of every 15-minute interval, there will be a single person at one of these positions. Note that there is no position 4, since if a person made it across the tightrope, then someone else would assume position 0 at the beginning of the tightrope. The transition matrix for this DTMC is $$P = \begin{bmatrix} p&q&0&0\\ p&0&q&0\\ p&0&0&q\\ 1&0&0&0 \end{bmatrix}$$ where $q:=1-p$.

Let $T_n$ be the random variable that records the time at which the $n$th person has fallen between positions 2 and 3.

Let $N(t)$ be the random variable recording the number of people that fall between positions 2 and 3 before time $t$.

Then one result we have from renewal theory is that

$$\lim_{n\to \infty} \frac{T_n}{n} = \mu \iff \lim_{t \to \infty} \frac{N(t)}{t} = \frac{1}{\mu}$$

$\mu$ can be thought of as the long-run average amount of time it takes for a person to fall between position 2 and 3. So the value we seek to find is $\frac{1}{\mu} = \frac{1}{\mathbb{E}[X]}$, where $X$ is defined below.

Let $X$ be the random variable recording the time it takes for the first person to fall between positions 2 and 3. We will calculate $\mathbb{E}[X]$.

Let $L_1$ be the random variable recording the time until the 2nd person starts walking across the tightrope (measured in hours).

\begin{align*} \mathbb{E}[X] &= \underbrace{\mathbb{E}\left[X\ \big| \ L_1 = \frac{1}{4}\right]}_{\text{starts over}} p + \underbrace{\mathbb{E}\left[X\ \big| \ L_1 = \frac{1}{2}\right]}_{\text{starts over}} qp + \underbrace{\mathbb{E}\left[X\ \big| \ L_1 = \frac{3}{4}\right]}_{\text{success}} q^2p + \underbrace{\mathbb{E}\left[X\ \big| \ L_1 = 1\right]}_{\text{starts over}} q^3 \\ &= (1/4 + \mathbb{E}[X]) p + (1/2 + \mathbb{E}[X])qp + 3/4 q^2 p + (1 + \mathbb{E}[X]) q^3 \\ &= \mathbb{E}[X] (p + qp + q^2p +q^3) + \frac{1}{4} p + \frac{1}{2} qp + \frac{3}{4} q^2 p + q^3 \end{align*}

Solving for $\mathbb{E}[X]$, we get $$\mathbb{E}[X] = \frac{\frac{1}{4} p + \frac{1}{2} qp + \frac{3}{4} q^2 p + q^3}{1- (p + qp + q^2p +q^3)}$$

And the long-run rate is $\frac{1}{\mathbb{E}[X]}$.

Answer 2

All times in this answer are measured in units of quarter hours.

The expected value of the time $T$ a person blocks the tightrope is

$$\sum_{k=0}^3P(T\gt k)=\sum_{k=0}^3(1-p)^k=\frac{1-(1-p)^4}p\;.$$

Thus the long-run rate at which people enter the tightrope is the reciprocal,

$$ \frac p{1-(1-p)^4}\;. $$

The probability for any given person to fall between positions $2$ and $3$ is $(1-p)^2p$.

Thus, the long-run rate at which people fall between positions $2$ and $3$ is

$$ \frac{(1-p)^2p^2}{1-(1-p)^4}\;. $$

Here's a plot.

(Multiply this by $4$ to get the rate per hour.)