Let $N(y) = \sum_n1_{\{X_n=y\}}$ be the number of visits to state $y$. Why is $E_xN(y) = \sum_{n=1}^{\infty} p^n(x,y)$?
$E_xN(y) = \sum_{n=1}^{\infty} P(N(y)\geq n)$ and I am confused a bit because $N(y)$ is the number of visits and $p^n(x,y)$ corresponds to one visit but in $n$ steps. Thanks.
Because $N(y)=\sum_{n=1}^\infty 1(X_n=y)$ and then
$$E_x[N(y)] \\ =E \left [\sum_{n=1}^\infty 1(X_n=y) \mid X_0=x \right ] \\ =\sum_{n=1}^\infty E[1(X_n=y) \mid X_0=x] \\ =\sum_{n=1}^\infty P[X_n=y \mid X_0=x] \\ =\sum_{n=1}^\infty p^n(x,y).$$
The main intuitive point is that the expected number of visits to $y$ at time $n$ is $p^n(x,y)$. The rest is just using linearity of expectation.