Relation between the expected number of visits to a state and reachability in a Markov chain

294 Views Asked by At

Let's consider a discrete time Markov chain $X_n$. Let $R_{ij} = \sum_{n=0}^\infty \mathbb{1}_{\{X_n= j | X_0 = i\}}$ be the number of visits to $j$ starting from $i$, and let $f_{ij}$ be $\text{Prob}\{X_n = j \text{ for some } n \ge 0 | X_0 = i\}$. The state $i$ and $j$ are both transient. The textbook with which I'm teaching myself states that $ER_{ij} = \mathbb{1}_{\{i = j\}} + f_{ij} ER_{jj}$ where $E$ is the expectation value operator, without any proof or explanation. I want to drive it. Here is my approach. Firstly $R_{ij}$ can be written as $$ R_{ij} = \begin{cases} \mathbb{1}_{\{i = j\}}+R_{jj} &\text{if it visits $j$ at least once,}\\ \mathbb{1}_{\{i = j\}} &\text{otherwise}. \end{cases} $$ The first case occurs with probability $f_{ij}$ and the other has the probability $1 - f_{ij}$. So, $$ \begin{eqnarray} ER_{ij} &=& f_{ij} E(\mathbb{1}_{\{i = j\}} + R_{jj}) + (1-f_{ij})E\mathbb{1}_{\{i=j\}}\\ &=&\mathbb{1}_{\{i=j\}} + f_{ij}ER_{jj}. \end{eqnarray} $$ Here I believe that the first line should be right. But I feel like that some intermeidate steps could be made to drive the first line clearly although I failed to write the steps. I'd appreciate it if someone drives it or points a reference.