Given a finite irreducible Markov chain with transition matrix $P$ and stationary distribution $\pi$, how would you prove the following for any initial distribution vector $v$?
$$\lim_{n \to \infty} \frac{v\sum\limits_{k=1}^n P^k}{n} = \pi$$
Inside the limit is the expected number of times we visit each state over the first $n$ time steps.
This is a special case of the elementary renewal theorem (see here), but I'm hoping for a direct proof from the limit above.
If I assume the limit exists, I can prove that it satisfies $\pi = \pi P$, but how can I prove the limit exists?
A similar question was asked here, but existence of the limit was not shown.
If $P$ is irreducible, and if $\pi=\pi P$ is a stationary distribution, then all components of $\pi$ are positive.
From this it follows that $P$ has at most one stationary distribution. For if $\alpha=\alpha P$ and $\beta=\beta P$ are distinct stationary distributions, then $\gamma=(\beta-m\alpha)/(1-m)$ is also a stationary distribution, where $m=\min_i \beta_i/\alpha_i$. But if $j$ is such that $m=\beta_j/\alpha_j$, then $\gamma_j = 0$, contradicting the previous result.
The set of probability vectors is compact, so the sequence $s_n=v\sum_{i=1}^nP^i / n$ has convergent subsequences. Let $\alpha$ be the limit of such a subsequence:$$ \alpha=\lim_{i\to\infty} s_{n_i}.$$ It is easy to check that $\alpha$ is stationary, since $$ \alpha P-\alpha =\lim_{i\to\infty} \frac{v (P^{n_i+1}-I)}{n_i}=0.$$ That is, any limit point of $s_n$ is stationary. But we saw $P$ has at most one stationary distribution. Since all the limit points of subsequences of $s_n$ are the same, the sequence $s_n$ itself converges to a stationary distribution.