Suppose I have a Markov Chain with state space $\mathbb{Z}$ with
$\mathbb{P}(X_n=X_{n-1}+1|X_{n-1})=\lambda$, $\mathbb{P}(X_n=X_{n-1}-1|X_{n-1})=\mu$
where $\lambda+\mu=1$, $\lambda,\mu>0$ and $\lambda\neq\mu$. How do I compute the expected number of visits to a state $n$?
MC=Markov Chain.
Let $$f_{jj}=P( \text{MC visits(in finite time) state}\;j \mid X_0=j).$$
Notice that given the Markov property once the MC visits state $j$ either returns to this state with probability $f_{jj}$ or leaves it forever with probability $1-f_{jj}$. Let $A$ be this last event with $P(A)=1-f_{jj}$.
Let $N_j$:=the number of visits of the MC to the state $j$= $\sum_ {n=1}^{\infty} 1 (X_n=j)$.
Then, $$P(N_j=k\mid X_0)=[1-P(A)]^{k-1}P(A)=f_{jj}^{k-1}(1-f_{jj})$$
This means that $$N_j\mid X_0 \sim \text{Geometric}(p=1-f_{jj})$$
The expected number of visits is $E(N_j\mid X_0)=\frac{1}{1-f_{jj}}$
This is finite when $f_{jj}<1$. A non-symmetric random walk the chain abandons state $j$ with positive probability $1-f_{jj}$ so the expectation is finite.
In simple terms a state is recurrent when beginning at this state the chain at some point revisits it with probability 1. A simple random walk is recurrent only when $\mu=\lambda=1/2$