Billingsley Exercise 8.8 (Markov Chains)

Question

I am studying from Billingsley and would like some hints on the following exercise.

Suppose $S = \{0,1,2,...\}$, $p_{00} = 1,$ and $f_{i0} > 0$ for all $i$.

Here, $S$ represents the state space, $$f_{ij} = P_{i} \left(\bigcup_{n=1}^{\infty} [X_{n} = j]\right)$$ and $P_{i}(A) = P[A\mid X_{0} = i]$.

Show that $$P_{i}\left(\bigcup_{j = 1}^{\infty}[X_{n} = j \;\; \text{i.o.}]\right) = 0$$ for all $i$.

Answer 1

Denote by

$$\tau_j := \inf\{n \geq 1; X_n = j\}$$

the hitting time and define, iteratively,

$$\tau_j^k := \inf\{n > \tau_j^{k-1}; X_n = j\}$$

for $k \geq 2$.

---

Hints:

1. It suffices to show that $P_i(X_n = j$ i.o.$)=0$ for all $j \geq 1$.
2. Show that $$P_j(\tau_0<\infty) \leq P_j\left( \left\{ X_n = j \, \, \text{i.o.} \right\}^c \right).$$ Conclude that $P_j(X_n = j \, \text{infinitely often})<1$.
3. Set $p:= P_j(\tau_j< \infty)$. Using the (strong) Markov property, show that $$P_j(X_n = j \, \text{for at least $k$ times}) = P_j(\tau_j^k <\infty) = p^k.$$ Conclude that $$P_j(X_n = j \, \text{infinitely often}) \in \{0,1\}$$ depending whether $p<0$ or $p=1$.
4. Combine step 2+3 to deduce that $P_j(X_n = j \, \text{infinitely often}) =0$.
5. By the Markov property, $$P_i(X_n = j \, \text{infinitely often}) \leq P_j(X_n = j \, \text{infinitely often}).$$ Now the claim follows from step 3.