Could someone please point me in the right direction for solving this problem? I am new to Markov chain theory and am confused about how to start, thanks.
Consider a Poisson process $\{N(t): t\geq 0\}$ with parameter $\lambda$, which can be interpreted as a continuous-time Markov chain. Find the corresponding matrix $P(t)$.
The Poisson process $N(t)$ has infinitesimal generator $$ Q_{ij} = \begin{cases} \lambda,& j=i+1\\ -\lambda,& j=i\\ 0,& \text{otherwise}. \end{cases} $$ We can show by induction that $$ Q_{ij}^{(n)} = \begin{cases} \binom n{j-i} \lambda^n (-1)^{n-(j-i)},& 0\leqslant j-i\leqslant n\\ 0,& \text{otherwise}. \end{cases} $$ The transition probabilities are given by $P(t) = e^{tQ}$, and so for each pair of nonnegative integers $j\geqslant i$ we have \begin{align} P_{ij}(t) &= \sum_{n=j-i}^\infty \frac{t^n}{n!}Q_{ij}^{(n)}\\ &=\sum_{n=j-i}^\infty \frac{t^n}{n!}\binom n{j-i} \lambda^n (-1)^{n-(j-i)}\\ &=\sum_{n=j-i}^\infty \frac{t^{j-i}}{(j-i)!}\lambda^{j-i}\frac{t^{n-(j-i)}}{(n-(j-i))!}(-\lambda)^{n-(j-i)}\\ &= \frac{(\lambda t)^{j-i}}{(j-i)!}\sum_{n=j-i}^\infty \frac{(-\lambda t)^{n-(j-i)}}{(n-(j-i))!}\\ &=\frac{(\lambda t)^{j-i}}{(j-i)!} \sum_{k=0}^\infty \frac{(-\lambda t)^k}{k!}\\ &=\frac{(\lambda t)^{j-i}}{(j-i)!} e^{-\lambda t}. \end{align} As expected, the probability of transitioning from state $i$ to state $j$ (with $j\geqslant i$) in $(0,t]$ is the probability of there being $j-i$ arrivals in $(0,t]$.