So, here's a question that has propped up in my research. Suppose that we have at our disposal $K$ independent homogeneous Markov chains evolving on a common, finite state space $S$. At each time instant $t=0,1,2,\ldots$, only one out of the $K$ Markov chains may be selected and its state may be observed. During this time, the unobserved Markov chains continue to evolve.
As a consequence of the last point mentioned above, there is a certain 'delay' associated with observing each Markov chain. Let $\tau$ be a stopping time until when the Markov chains are sampled one at a time in a sequential manner. We shall denote by $N_a(\tau,d,i)$ the number of times Markov chain $a$ is selected when its delay is $d$ and it was observed in state $i\in S$ the previous time. Here, $d\leq \tau$ and $a\in\{1,2,\ldots,K\}$.
I would like to compute the following expectation: \begin{equation} E\bigg[\sum\limits_{a=1}^{K}\sum\limits_{d=1}^{\tau}\sum\limits_{i\in S}N_a(\tau,d,i)\,c(a,d,i)\bigg], \end{equation} where $c(a,d,i)$ are positive constants for each $a,d,i$ triple.
While the summation terms with respect to $a$ and $i$ are finite sums, there is no problem with pushing the expectation inside these summations. However, there is $\tau$ in the upper limit of the summation over $d$, and it looks like I need a Wald type identity to simplify this summation and expectation. However, I do not know of any such identity in the current context of Markov chains. Moreover, the collection of random variables $\{N_a(\tau, d,i)\}_{1\leq d\leq \tau}$ is not really independent.
Any help in this direction would be greatly appreciated.