Discrete type Markov chains Long run proportions?

Question

A certain missile has a guidance control system. Course correction signals form a sequence ${X_n; n \geq 0}$ where $X_n$ is the state of the system after the nth signal is sent and received. The state space for X is:

0: No correction required,

1: Minor correction required,

2: Major correction required,

3: Abort and self-destruct.

Suppose that {Xn, n \geq 0} can be modeled as a discrete-time Markov chain with one-step transition matrix

If the system is initially in state 1, find the probability that the missile will self-destruct.

Hi, looking at this question, it looks like we need to find the long run proportion for state 0. Since we see that state 0 and 3 are reccurent. This means that $\pi_1=0$ and $ \pi_2 = 0$ as they are transient states. I tried finding long run proportiion for state 0 and 4 by $\pi = \pi P$ but cannot get the answer.

Answer 1

This is an absorbing Markov chain, so there is no unique limiting distribution $\pi$ such that $$\pi(j) = \lim_{n\to\infty}\mathbb P(X_n=j) $$ for each state $j$. Re-order the states $(1,2,0,3)$ to obtain $$ P = \pmatrix{0&1/6&5/6&0&\\5/6&0&0&1/6\\0&0&1&0\\0&0&0&1} = \pmatrix{Q&R\\0&I_2}, $$ where $Q$ is the substochastic matrix corresponding to transitions between transient states and $R$ is the substochastic matrix corresponding to transitions from transient states to absorbing states. Let $N$ the matrix be defined by $$N_{ij} = \sum_{n=0}^\infty \mathsf 1_{\{X_n=j\ \mid\ X_0=i\}}, $$ then $N=\sum_{n=0}^\infty Q^n$. Since $Q$ is a substochastic matrix, this series converges and is equal to $(I-Q)^{-1}$. We compute $$ (I-Q)^{-1}=\pmatrix{1&-1/6\\-5/6&1}^{-1} =\pmatrix{36/31&6/31\\30/31&36/31}, $$ and so the expected number of transitions until absorption when starting in transient state $i$ is the $i^{\mathsf{th}}$ entry of $$ N\cdot\mathbf 1 = \pmatrix{36/31&6/31\\30/31&36/31}\pmatrix{1\\1}=\pmatrix{42/31\\66/31}. $$ The probability of being absorbed in state $j$ when starting in state $i$ is the $(i,j)$-entry of $$ NR = \pmatrix{36/31&6/31\\30/31&36/31}\pmatrix{5/6&0\\0&1/6}=\pmatrix{30/31&1/31\\25/31&6/31}. $$

Answer 2

Ok, I don't know the best way to approach this problem, but a possible approach follows.

The system is initially ($n=0$) in state 1, $p = (0,1,0,0)$ where $p_i$ is the probability of state $i$, $i = 0,\ldots , 3$.

Using the given transition probabilities, for $n=1$, $ p=(5/6, 0, 1/6, 0) $ $$\text{For } n = 2, \quad p = (5/6, 5/36, 0,1/36 ). $$

$$\text{For } n = 4, \quad p = \left(\frac{5}{6}+\frac{5}{6}\frac{5}{36}, \left(\frac{5}{36}\right)^2, 0,\frac{1}{36} + \frac{1}{36}\frac{5}{36} \right). $$

$$\text{For } n = 2t, \quad p = \left(\frac{5}{6}+\frac{5}{6}\frac{5}{36}+\cdots+\frac{5}{6}\left(\frac{5}{36}\right)^{t-1}, \left(\frac{5}{36}\right)^t, 0,\frac{1}{36} + \frac{1}{36}\frac{5}{36}+\cdots +\frac{1}{36}\left(\frac{5}{36}\right)^{t-1} \right) $$

In the limit, the long run probability of state 3 is $$ \frac{1}{36}\sum_{t=0}^{\infty} \left(\frac{5}{36}\right)^t = \frac{1}{31} $$