Let $(X_k)_k$ be a Markov chain with values in a countable set E and having a transition matrix $Q$
Let $k \in \mathbb{N}$ and $v \in E.$ Prove that $E[Q^k(X_{k+1},v)|X_0,...,X_k]=Q^{k+1}(X_k,v)$ a.s.
We note that $E[Q^k(X_{k+1},v)|X_0,...,X_k]=\sum_{u \in E}Q^{k}(u,v)P(X_{k+1}=u|X_0,...,X_k)=\sum_{u \in E} Q^k(u,v)Q(X_{k},u).$
How can we continue to obtain the result? Is there other way to prove this?