Markov chain transient state

Question

I have got a Markov Chain as below

$\begin{bmatrix}q & p & 0 & 0 &0\\0 & p &q & 0 &0\\0&p&0&q&0\\q&0&0&0&p\\0&0&0&0&1\end{bmatrix}$

I am asked to classify the class and if they are transient. The answer from my lecturer is {0,1,2,3} is a open class so that its transient; and {4} is closed so that its persistent. What i am struggling is, for the class {0,1,2,3}, for example taking state 1, it can get back to state 1 in the next step, otherwise it can go to state 2, then it either back to state 1 or go to state 3, then it either go to state 4 which the matrix will terminate, or it can go to state 0 which can go back to state 1. So i can conclude that if we start in state 1, there are possibilities that it will go back to state 1 so that {0,1,2,3} which include state 1 shouldn't be transient.

Are there any mistake in my argument or i have mess up the definition of a transient state?

Many thanks

Answer 1

A state is transient if there's a nonzero proability that, starting there, that state is never visited again. State 1 in your chain is transient, because from state 1 you can go (with probability $q$) to state 2, then (with probability $q$) to state 3, then (with probability $p$) to state 4, after which you will never return to state 1; so if you start at 1 the probability of never returning to 1 is at least $pq^2$ which is bigger than 0.

I think you may be thinking that "transient" means "nonzero probability of returning", but that's not right: "recurrent" means "zero probability of not returning" and "transient" means "nonzero probability of not returning" or "not-1 probability of returning".

Answer 2

For a finite state Markov chain, you can think in the following way:

1) If it is possible to reach $y$ from $x$ and it is possible to reach $x$ from $y$, then $x$ and $y$ are in the same group. From your line of reasoning, $0,1,2$ and $3$ are all inthe same group. Note that, although it is possible to reach $4$ from $3$, it is not possible to reach $3$ from $4$. This is why $4$ is not in the same group. As a result, you have the groups $\{0,1,2,3\}$ and $\{4\}$.

2) If there exists $x_1$ in a group $G_1$ and $x_2$ in a group $G_2$ such that it is possible to reach $x_2$ from $x_1$, then $G_1$ is transient. For example, it is possible to reach $4$ from $3$. Therefore, $\{0,1,2,3\}$ is transient.

3) If $G$ is a group and, for every state $x$ in $G$, it is only possible to reach a state in $G$ from x, then $G$ is recurrent. Since it is only possible to reach 4 from 4, $\{4\}$ is recurrent.