Help on some results with Markov chains

Question

I currently have a markov chain represented by the following matrix: $$\left[\begin{array}{ccc} 0&1&0 \\ .99&0&.01 \\ 0&0&1 \end{array}\right],$$ i.e. a row-stochastic matrix, with uniform probability on the initial state distribution. I am trying to figure out the probability that the hitting time for state $3$ is finite. I am not completely certain about how to go about these calculations, but I would imagine this may have something to do with the absorbent state qualities of state $3$. Any suggestions on how to go about this kind of problem?

Answer 1

Let $$\tau=\inf\{n\geqslant 0: X_n=3\} $$ and $$\tau_i = \inf\{n\geqslant 0: X_n=3\mid X_0=i\}.$$ Conditioning on $X_1$ yields the system of equations \begin{align} \mathbb E[\tau_1] &= 1 + \mathbb E[\tau_2]\\ \mathbb E[\tau_2] &= 1+ \frac{99}{100}\mathbb E[\tau_1]\\ \mathbb E[\tau_3] &= 0, \end{align} and hence $\mathbb E[\tau_1]=200$, $\mathbb E[\tau_2]=199$. It follows that $$\mathbb E[\tau] = \frac13(200 + 199) = 133<\infty. $$

As @ByronSchmuland pointed out, this technique requires the assumption that $\tau$ is integrable. So we may instead compute the distribution of $\tau$ directly. Clearly $\mathbb P(\tau_3=0)=1$, and further \begin{align} \mathbb P(\tau_1=2)&=P_{12}P_{23} = \frac1{100}\\ \mathbb P(\tau_2=1)&=P_{23}=\frac1{100}. \end{align} Since $P_{11}=P_{22}=0$, it follows that $$\mathbb P(\tau_1=2(n+1))= (P_{12}P_{21})^nP_{12}P_{23} = \left(\frac{99}{100}\right)^n\frac1{100}$$ and similarly $$\mathbb P(\tau_2=2n+1) = (P_{21}P_{12})^nP_{23} = \left(\frac{99}{100}\right)^n\frac1{100} $$ for $n=1,2,\ldots$. Therefore \begin{align} \mathbb E[\tau_1]&=\sum_{n=0}^\infty 2(n+1)\left(\frac{99}{100}\right)^n \frac1{100}\\ &=\frac1{50}\sum_{n=0}^\infty (n+1)\left(\frac{99}{100}\right)^n\\ &=\frac{100^2}{50}\\ &=200, \end{align} and similarly $$\mathbb E[\tau_2] = \sum_{n=0}^\infty(2n+1)\left(\frac{99}{100}\right)^n \frac1{100}=199. $$ We conclude that \begin{align} \mathbb E[\tau] &= \sum_{i=1}^3 \mathbb E[\tau_i]\mathbb P(X_0=i)\\ &= \frac13(\mathbb E[\tau_1]+\mathbb E[\tau_2]+\mathbb E[\tau_3])\\ &= \frac13\left(200+199+0\right)\\ &= 133. \end{align}

Answer 2

State 3 is absorbing, and from state 1 the chain always moves to state 2. So to never hit state 3, the chain must either (A) start in state 1 and bounce back and forth with state 2, or (B) start in state 2 and bounce back and forth with state 1. Both of these events have probability zero: Event $A$ implies that the chain starts at $1$ and the next $2n$ visits are $(2,1,2,\ldots,1,2,1)$; this has probability $\frac13(1)^n(.99)^n$ (after appropriate grouping of terms), so $$ P(A) \le \frac13(.99)^n.$$ This holds for every $n$, hence $P(A)=0$. A similar argument shows $P(B)=0$. Therefore the probability is zero that the chain never hits state 3, and the prob is one that the hitting time for state 3 is finite.