What is the expected number of steps until we first reach the rightmost state in this markov chain?

890 Views Asked by At

enter image description here

The transition probability is 0.5 everywhere (except for last and first states with 1).

I've tried the following: Marking $T_j$ as the number of steps that took to first reach state $s_j$: $\mathbb{E}\left[T_{j}\right]=\mathbb{E}\left[\mathbb{E}\left[T_{j}\rvert T_{j-1}\right]\right]=\mathbb{E}\left[\underset{\text{Straight from j-1 to j}}{\underbrace{0.5\left(T_{j-1}+1\right)}}+\underset{\text{Reset from j-1 to 1}}{\underbrace{0.5\cdot x}}\right]$

I'm not sure if my initial thought is correct and even if it is, I'm not sure what's $x$ in the last row or how to continue.

If it helps, I've calculated the transition probability matrix:

$P=\begin{pmatrix}0 & 1 & 0 & 0 & \cdots & 0\\ 0.5 & 0 & 0.5 & 0 & \cdots & 0\\ 0.5 & 0 & 0 & 0.5 & \cdots & 0\\ & & \vdots & \vdots\\ 0.5 & 0 & 0 & \cdots & 0 & 0.5\\ 1 & 0 & 0 & \cdots & 0 & 0 \end{pmatrix}$

1

There are 1 best solutions below

2
On

You're close. Let $E_j$ be the expected number of steps to get to state $j$ from state $1$. Then $$E_j=E_{j-1}+\frac12(1)+\frac12(1+E_j)\implies E_j=2E_{j-1}+2$$

In order to get to state $j$, we must first get to state $j-1$. There's no reason to multiply by $\frac12$ here. Then half the time, we finish in one more transition, and half the time we begin all over.

In response to the OP's comment, I've recast it in terms of conditional expectation.

$$\begin{align} E(T_j)&=\sum_iE(T_j|T_{j-1}=1)\Pr(T_{j-1}=i)\\ &=\frac12\sum_i(i+1)\Pr(T_{j-1}=i)+\frac12\sum_i(i+1+E(T_j))\Pr(T_{j-1}=i)\\ &=1+E(T_{j-1})+\frac12\sum_iE(T_j)\Pr(T_{j-1}=i)\\ &=1+E(T_{j-1})+\frac12E(T_j) \end{align}$$