So the problem is: Let $X$ be a Markov chain with state space $E = ${a,b} and transition matrix $$p=\begin{bmatrix}0.4 &0.6 \\1 & 0 \end{bmatrix}$$
and suppose that a reward of $g(i,j)$ units is received for every jump from $i$ to $j$ where $$g=\begin{bmatrix}3 &2 \\-1 & 1 \end{bmatrix}$$
Find $$\lim_{n\to \infty}\frac 1{n+1}\sum_{m=0}^{n}g(X_m, X_{m+1})$$
The answer is $\frac 9{8}$.
But there are no steps to the answer. I really appreciate if you could show me how to get to the answer, thanks!
Suppose $X_0 = b$. Let $W_0=0$ and $$W_{n+1} = \inf\{m>W_n: X_n=b\}. $$ The $W_n$ are independent with common distribution $$\mathbb P(W_1=k) = \left(\frac25\right)^{k-2}\frac35,\ k=2,3,\ldots $$ and so $\{W_n:n=1,2,\ldots\}$ is a renewal process. Let $R_n$ be the reward accumulated between $W_{n-1}$ and $W_n$. Then the $R_n$ are also independent with common distribution $$\mathbb P(R_1 = 1+3k) = \left(\frac25\right)^k\left(\frac35\right), k=0,1,2,\ldots$$ The mean interrenewal time is $$\mathbb E[W_1] = \sum_{k=2}^\infty k\ \mathbb P(W_1=k) = \sum_{k=2}^\infty k\left(\frac25\right)^{k-2}\frac35 = \frac83 $$ and the mean reward accumulated $$\mathbb E[R_1] =\sum_{k=0}^\infty (1+3k)\ \mathbb P(R_1=1+3k) = \sum_{k=0}^\infty(1+3k)\left(\frac25\right)^{k-2}\frac35 = 3. $$ It follows from the renewal reward theorem that both the long-run time average reward rate and the long-run average expected reward are given by $$\frac{\mathbb E[R_1]}{\mathbb E[W_1]}=\frac3{8/3}=\frac98. $$