With the corona crisis education is now online. Including poorly recorded lectures that use a robot camera to follow around the lecturer when he walks around the room. Due to this I have been struggling with certain concepts about Markov chains.
I have asked some questions about the following instruction question, but our university forum does not support latex formatting. Note: this is not homework, this is a question to prepare oneself for doing the homework that comes later this week. I want to be prepared.
A three-state Markov chain has the transition matrix: $$P = \left(\begin{array}{ccc} p & 1- p & 0 \\ 0 &0 &1 \\ 1-q & q & 0 \end{array} \right)$$ Where $p, q \in (0, 1)$.
Task: Show that state $1$ is persistent.
I do not understand how one goes about to show persistence? I already observe that due to the nature of the problem, one can only go back to state $1$ via the other states in an odd number of steps, so if we let $T_{11}$ be the time/number of steps it take to get from $1$ back to $1$, assuming the previous state was not $1$, we can get scenarios like: $$P(T_{11}=1) = p$$
$$P(T_{11}=3) = (1-p)(1-q)$$
$$P(T_{11}=5) = (1-p)(1-q) q$$
$$P(T_{11}=7) = (1-p)(1-q) q^2$$
$$P(T_{11}=9) = (1-p)(1-q) q^3$$
As has been pointed out in a comment, there are other paths via which the system can return to the first state. Trying to enumerate them can give you an idea of what’s going on, but is quite prone to errors of omission, as here.
A state is persistent iff the probability of the system returning to that state when starting from it is $1$, so let’s compute this probability directly. Let $p_n$ denote the probability of returning to state $1$ from state $n$. From the transition matrix we then obtain the following system of equations: $$p_1=p+(1-p)p_2 \\ p_2=p_3 \\ p_3 = (1-q)+qp_2.$$ This system is easily solved for $p_1$ via back-substitution.