Fix a number p with $0 < p < 1$. Consider the Markov chain on the non-negative integers $\{0, 1, 2, . . .\}$ whose transition probabilities are given by $P_{n,n+1} =p, P_{n,0} =1−p$, forall $n≥0$.
(a) Is this Markov chain irreducible? Explain.
(b) Is the state 0 periodic or aperiodic? Explain.
(c) Suppose that the Markov chain is initially in state $0$, and let $T_0$ denote the time of first return to $0$ (i.e., $T_0$ is the smallest value of $n > 0$ such that $X_n = 0$, if such a value exists, and otherwise $T_0 = ∞$). Determine the probability mass function of $T_0$.
(d) What exactly does it mean to say that 0 is a recurrent state?
(e) Prove that 0 a recurrent state.
This is a past final exam from my university. I don't have much confidence on discrete Markov Chains. Can you please verify my answer for part (a), (b), (c), and provide me with some enlightenment on part (d), (e)?
My answer:
(a) Yes, because all states communicate with each other.
(b) Aperiodic. Because it can come back to itself in one step therefore, having period of 1.
(c) $\pi_0 = \pi_0 p_{0,0} + \pi_1 p_{1,0} + \pi_2 p_{2,0} + \cdots $
$\pi_0 = (1-p)( \pi_0 + \pi_1 + \pi_2 + \cdots )$
$\pi_0 = (1-p)(\sum_{i \geq 0} \pi_i )$
$\therefore \pi_0 = (1-p), T_0 = \frac{1}{1-p}$
(d) I know by definition: $\lim_{n \rightarrow \infty} p(X_n=0) = 1$
(e) Should I say that because $T_0$ is finite and therefore, it is recurrent?
I was inspired by this answer to solve (e).
Let $\tau_{00} = \inf \{ n>0 : X_n = 0 | X_0=0 \}$. Then $\tau_{00} = 1$ with probability $1-p$ and $\tau_{00} > 1$ with probability $p$. In the second case, $\tau_{00} = 1 + T$ where $T$ has the geometric distribution with success probability $1-p$. Therefore, $E[\tau_{00}] = 1 + \frac{1}{1-p} < \infty$. Therefore, it is recurrent.