I am considering a Markov chain on S = {1, . . . , 21} with transition matrix given by: $ 1 =p_{1,2} = p_{2,1}= p_{13,14} = p_{18,14} = p_{15,16} = p_{16,3} = p_{17,16}\\ 1/2 =_{p5,5} =p_{5,7}=p_{7,7} =p_{9,7} =p_{9,9} =p_{12,13} =p_{12,18}\\ 1/4 = p_{7,5} = p_{7,9}\\ p = p_{3,2} = p_{4,5}= p_{10,9} = p_{11,1} = p_{14,20}\\ p/2 = p_{6,5} =p_{6,7}=p_{8,7} =p_{8,9} \\ 1−p = p_{3,4} =p_{4,6}=p_{6,8} =p_{8,10} =p_{10,11} =p_{11,12}\\ (1 − p)/2 = p_{14,15} = p_{14,17}\\ 1/2 =p_{19,19} = p_{19,20} = p_{20,19} = p_{20,21} = p_{21,21} = p_{21,20} $
I have founded the communication classes and determined for each class if they are recurrent or transient. Now I want to find the limit
$\lim_{n \to \infty}P(X(n)=20)$
where the initial distribution is given bu $P(X(0)=11)=1$. Here state 20 is in a closed recurrent class and state 11 is in a transient class.
I think I have to consider the distribution of the time of the first jump to state 20: $\tau_{20}=\inf \lbrace n>0 \vert X(n)=20 \rbrace$, but can get any further. Can anyone help me or give a hint?
The relatively high number of states is only there to force you to understand the structure of the paths the Markov chains can follow to reach state $20$ starting from state $11$. Thus, a mandatory prologue to the proof is to draw a picture of the states and the transitions of the Markov chain...
Once this is done (and if you did not do it, the proof below shall be impossible to follow while, if you did do it, it should be nearly trivial), one sees that the class $$\mathtt C=\{19,20,21\}$$ is closed and that the only way to reach $\mathtt C$ directly starting from $11$ is through the steps $$11\stackrel{1-p}{\longrightarrow}12\stackrel{1}{\longrightarrow}\{13,18\}\stackrel{1}{\longrightarrow}14\stackrel{p}{\longrightarrow}20,$$ which have probability $$p_{11,12}p_{14,20}=p(1-p).$$ This is so, because, after some step $11\stackrel{p}{\longrightarrow}1$, one stays in the closed class $$\mathtt D=\{1,2\},$$ hence one never hits $\mathtt C$.
The other option is to choose the step $14\stackrel{1-p}{\longrightarrow}\{15,17\}$. Then, a careful tracking of the paths with positive probability reveals that one always reaches $3$. The transition $3\stackrel{}{\longrightarrow}2$ leads to $\mathtt D$, hence it forbids the visits to $\mathtt C$. The other transition is $3\stackrel{1-p}{\longrightarrow}4$. Starting from $4$, one reaches either $11$ or the class $$\mathtt F=\{5,7,9\}.$$ This class is closed hence the only way to visit $\mathtt C$, starting from $4$, is to avoid $\mathtt F$ to reach $11$ and then to hit $\mathtt C$ starting anew from $11$.
This decomposition shows that $$P_{11}(\text{hits}\ \mathtt C)=p(1-p)+P_{11}(\text{hits}\ 4)P_4(\text{hits}\ 11)P_{11}(\text{hits}\ \mathtt C).$$ where $$P_{11}(\text{hits}\ 4)=p_{11,12}(p_{14,15}+p_{14,17})p_{3,4}=(1-p)^3.$$ To hit $11$ starting from $4$, the only possible path is $$4\stackrel{1-p}{\longrightarrow}6\stackrel{1-p}{\longrightarrow}8\stackrel{1-p}{\longrightarrow}10\stackrel{1-p}{\longrightarrow}11,$$ hence $$P_4(\text{hits}\ 11)=(1-p)^4.$$ This yields $P_{11}(\text{hits}\ \mathtt C)$. Now, in the closed class $\mathtt C$, things are simple since the transitions are $$⟳19\rightleftarrows20\rightleftarrows21⟲,$$ all with probability $\frac12$, hence the class $\mathtt C$ is aperiodic with some unique stationary distribution $\pi_\mathtt C$, and one can check that $\pi_\mathtt C$ is uniform on $\mathtt C$.
Finally, all this proves that $$\lim_{n\to\infty}P_{11}(X_n=20)=P_{11}(\text{hits}\ \mathtt C)\cdot\pi_\mathtt C(20)=\frac{p(1-p)}{3(1-(1-p)^7)}.$$