convergence of nullrecurrent markov chain

Question

Hi guys!

At the moment I'm working on this proof. It's in a german book so hopefully you understand everything.

I understand everything in the picture without the use of the markov property at the end (where does he use it?).

Then in the author follows form the last equations at the picture that $\forall n\ge K\ \exists k=k(n) \in \{0,...,K\}$ such that $\mathbb{P}(X_{n-k(n)}=j)<\varepsilon$

I don't know why this holds. Can anybody help me?

Answer 1

The (simple) Markov property is used at the first equal sign after $1\geqslant\cdots$, to assert that, for each time $k$, the probability $P(X_k=j,\text{something happens after time $k$})$ is also $P(X_k=j)$ times $P_j(\text{same thing shifted by time $k$ happens})$.

Re your second question, introduce $\alpha=\inf\limits_{0\leqslant\ell\leqslant K}P(X_{n-\ell}=j)$ and note that the inequality ending with $\cdots\geqslant\frac2\varepsilon$ and the inequality starting with $1\geqslant\cdots$ together prove that $1\geqslant\sum\limits_{k=0}^KP_j(T_j\gt k)\alpha\geqslant\frac2\varepsilon\alpha$ hence $\alpha\leqslant\frac\varepsilon2$ and, indeed, there exists some $\ell$ in $\{0,1,\ldots,K\}$ such that $P(X_{n-\ell}=j)\leqslant\varepsilon$.