Why do we require positive recurrence for a Markov chain to have steady states?

94 Views Asked by At

Theorem 4.1 of the book Introduction to Probability Models (10th edition) by Sheldon Ross states that an "irreducible ergodic" Markov chain has limiting probabilities that exist.

And ergodic further means that it must be positive recurrent and aperiodic.

Why did he exclude null recurrent chains? Null recurrence means that the distribution of the number of time steps between visiting the same state doesn't have a mean (a Cauchy like distribution). What's wrong with that kind of distribution? How does that make the chain not have valid steady states?


An example of a null recurrent Markov chain is the one dimensional random walk (symmetric). But that is also not aperiodic (so periodic?). Perhaps we can "fix" that by adding a self-loop at the state $0$ (or even all the states). Does such a chain not have a steady state distribution?

1

There are 1 best solutions below

3
On

$\def\ed{\stackrel{\text{def}}{=}}$ Ross's theorem $4.1$ says a little more than that an irreducible ergodic chain "has limiting probabilities that exist". Here's the theorem as stated by Ross:

For an irreducible ergodic Markov chain $\ \lim_\limits{n\rightarrow\infty}P_{ij}^n\ $ exists $\color{red}{\text{and is independent of }\ i}\ .$ Furthermore, letting $$\pi_j=\lim_\limits{n\rightarrow\infty}P_{ij}^n, j\ge0,$$ then $\ \pi_j\ $ is the unique nonnegative solution of \begin{align} \pi_j&=\sum_{i=0}^\infty\pi_iP_{ij}, j\ge0,\\ \sum_{j=0}^\infty \pi_j&=1\tag{4.7}\label{eq}\end{align}

The conclusions of the theorem don't hold for any chain with null recurrent states. From equation (\ref{eq}) it follows that there's at least one state $\ s\ $ for which $\ \pi_s>0\ $. But if $\ i\ $ is a null recurrent state, then $$ \lim_\limits{n\rightarrow\infty}P_{is}^n=0\ne\pi_s\ , $$ which would contradict the limit's independence of $\ i\ $.

If you drop the conclusion that the limit is independent of $\ i\ $ then you can weaken the hypotheses of the theorem by replacing "irreducible ergodic Markov chain" with "aperiodic Markov chain with exactly one positive recurrent class of states". If you also drop the conclusion that a nonnegative solution of the equations $\ \pi_j=\sum_\limits{i=0}^\infty\pi_iP_{ij}\ ,$$\ \sum_\limits{j=0}^\infty \pi_j=1\ $ be unique then you can further weaken the hypotheses by replacing the words "exactly one" with "at least one".