A state $i$ is recurrent, if and only there exists $n \geq 1$ s.t. $p_{ii}^{(n)} =1$?

Question

For a homogeneous discrete time Markov chain with transition matrix $p$, a state $i$ is recurrent, if and only there exists $n \geq 1$ s.t. $p_{ii}^{(n)} =1$?

I have it copied from somewhere in my notes, but I forgot about the source. I didn't write the chain is finite-state, so can I assume the above statement, if it is true, can apply to countable-state case?

Thanks and regards!

Answer 1

Not at all. Counterexample: simple random walk on any connected finite graph (with at least 3 vertices, say).

Answer 2

As a matter of definition, state $i$ is recurrent for the Markov chain $(X_n)_{n\ge 0}$ if and only if $$ P(X_n = i \hbox{ for some }n\ge 1 \,|\, X_0=i) =1. $$ It is well known that $i$ is recurrent if and only if $\sum_{n\ge 1} p^{(n)}_{ii} =\infty$.