Consider the discrete time Markov chain with the following transition graph:
Let $N(y)$ be the total number of visits to the state $y$, eventually excluding $X_0$. In symbols, $$N(y) = \sum_{n=1}^{\infty}1_{X_n = y}.$$ $(a)$ Calculate $E_3[N(3)].$
$(b)$ Calculate $E_4[N(4)].$
I feel quite confident that, for $(b)$, $E_4[N(4)] = \infty$, since the set $\left\{4,5\right\}$ is closed and irreducible. However, I'm unsure how to find the precise expectation $E_3[N(3)]$.
Of course, beginning at $3$, we have with probability $5/8$ that we never return to $3$ again. However, even assuming we go to $1$ or $2$, we will eventually end back at $3$ where we again have a $5/8$ probability of going to $4$ or $5$ and, thus, never returning to $3$ again.
I suppose we really want to know the expected number of times we return to $3$ without going to $4$ or $5$. How can I go about finding this number?
edit: I now believe I know the general idea: to find $E_3[N(3)]$ we must find $\sum_{n=1}^{\infty} p^{(n)}(3,3)$, where $p^{(n)}(3,3)$ is the n-step probability matrix for the chain. I'm not sure we ever learned a trick/formula to calculate this sum, however - what is a good way to go about this?