Consider the following theorem from the text "An Introduction to Stochastic Modeling" by Mark A. Pinsky and Samuel Karlin
Theorem 4.3 (a.) Consider a recurrent irreducible aperiodic Markov chain. Let $P_{ii}^{(n)}$ be the probability of enetering state i at the nth transition, $n=0,1,2, \cdots,$ given that $X_{0} =i$ (the initial state is $i$). By convention $P_{ii}^{(0)}=1$. Let $f_{ii}^{(n)}$ be the probability of first returning to state $i$ at the nth transition, $n=0,1,2 \cdots $, where $f_{ii}^{(0)} = 0$. Then, $lim_{n\rightarrow \infty}P_{ii}^{(n)} = \frac{1}{\sum_{n=0}^\infty nf_{ii}^{(n)}}=\frac{1}{m_i}$. (b.)Under the same conditions as in (a.), $lim_{n\rightarrow \infty}P_{ji}^{(n)} = lim_{n \rightarrow \infty}P_{ii}^{(n)}$ for all states $j$.
I am having a difficult time setting the proof for this up. I am tempted to do an $\epsilon$ proof for sequences here. My first attempt was doing this proof direct: $\lim_{n \rightarrow \infty} P_{ii}^{(n)} = \lim_{n \rightarrow \infty}\sum_{k=0}^n f_{ii}^{(n)}P_{ii}^{(n-k)} = \sum_{k=0}^\infty f_{ii}^{(n)}P_{ii}^{(n-k)}$. Then here I do not know where to go. Similar to the second part of this theorem. Please provide a detailed proof and references if necessary to this theorem.