Suppose we have an irreducible and recurrent discrete-time Markov chain with states over the finite set $\mathcal{X}$. Let $N_t (x)$ denote the number of visits to state $x$ up to time $t$. Let $\pi(x)$ denote the stationary distribution of this Markov chain, and let $X_1 \sim \pi(x)$, i.e., the Markov chain is initially at the stationary distribution. Is the following almost surely true?
$$ \lim_{T\to \infty}\frac{1}{T} N_T(x) = \pi(x) $$
Clearly we cannot directly use strong law of large numbers (at least its standard form), since $X_1, X_2, \dots$ are not independent. Intuitively it seems obvious but I could not find a way of proving it.
It's true. See for example Theorem 5.3 of this reference. The idea is that you can indeed apply the strong law of large numbers by splitting $N_T(x)/T$ into excursion times: considering how long it takes to come back to state $x$ when one has just reached state $x$. These excursion times are all independent of each other and hence the strong law applies.