Continuous Time Markov Chain to find probabilities

Question

I got part a, since the rows must sum to 1 $a = 2$ and $b = 6$. I am confused on b,c, and d. I would love some hints on how to approach these parts of the problem.

For b, I was thinking that $P(X_{4.7} = 1$ and $ X_1 = 1 | X_{1.3}=2$ and $X_0 =0) = P(X_{4.7} = 1, X_1 = 1 | X_{1.3}=2)$ by the Markov Property. Then I am stuck here and how to re-write the probability so that I can use the transition probabilities in the matrix.

For c, I was thinking that $P(X_6 =1) = P(X_6 = 1 | X_t = 2)$ for $t \in [0,6)$. Using $Q= P'(0)$, we need $q_{22}$ and $q_{21}$. So, $q_{21}$ at $t = 0$ is 2 and $q_{22}$ at $t = 0$ is -3 so $T(2,1) = \frac{q_{21}}{-q_{22}} = \frac{2}{3}$. I am not sure if this is a correct way of thinking about this problem or even accurate.

And lastly, for d, I am thinking to find Q and solve $\pi * Q = 0$ and $\pi_0 + \pi_1 + \pi_2 = 1$, but again, I am not sure if this is what the question is asking. I would appreciate any and all help. Thank you!

Answer 1

Hints

• You're on the right track to make use of the Markov property, but your equation isn't correct as it stands, because the time index for one of the conditioning events (viz. $\ X_{1.3}=2\ $) is greater than one of those in the conjunction whose conditional probability is being calculated (viz. $\ X_1=1\ $). It might be possible for someone very familiar with Markov chains to write down the correct equation without any calculation, but the only way I can do it is to expand out various conditional probabilities using their definitions: \begin{align} \hspace{1.5em}\mathbb{P}\big(X_{4.7}=1,&\,X_1=1\,\big|\,X_{1.3}=2,X_0=0\big)\\ &=\frac{\mathbb{P}\big(X_{4.7}=1,X_1=1,X_{1.3}=2,X_0=0\big)}{\mathbb{P}\big(X_{1.3}=2,X_0=0\big)}\\ &=\frac{\mathbb{P}\big(X_{4.7}=1,X_1=1,X_{1.3}=2,X_0=0\big)}{\mathbb{P}\big(X_1=1,X_{1.3}=2,X_0=0\big)}\\ &\hspace{1em}\times \frac{\mathbb{P}\big(X_1=1,X_{1.3}=2,X_0=0\big)}{\mathbb{P}\big(X_1=1,X_0=0\big)}\\ &\hspace{1em}\times\frac{\mathbb{P}\big(X_1=1,X_0=0\big)}{\mathbb{P}\big(X_0=0\big)}\\ &\hspace{1em}\times\frac{\mathbb{P}\big(X_0=0\big)}{\mathbb{P}\big(X_{1.3}=2,X_0=0\big)}\\ &=\mathbb{P}\big(X_{4.7}=1\,\big|\,X_1=1,X_{1.3}=2,X_0=0\big)\\ &\hspace{1em}\times \mathbb{P}\big(X_{1.3}=2\,\big|\,X_1=1,X_0=0\big)\\ &\hspace{1em}\times\frac{\mathbb{P}\big(X_1=1\,\big| \,X_0=0\big)}{\mathbb{P}\big(X_{1,3}=2\,\big|\,X_0=0\big)}\\ &=\mathbb{P}\big(X_{4.7}=1\,\big|\,X_{1.3}=2\big)\mathbb{P}\big(X_{1.3}=2\,\big|\,X_1=1\big)\\ &\hspace{1em}\times\frac{\mathbb{P}\big(X_1=1\,\big| \,X_0=0\big)}{\mathbb{P}\big(X_{1.3}=2\,\big|\,X_0=0\big)}\ , \end{align} where it is only in the last equation that all the conditional probabilities have the right form for the Markov property to be applicable.

• The definition of the transition matrix $\ \mathbf{P}\ $ is $$ \mathbf{P}_{ij}(t)=\mathbb{P}\big(X_{s+t}=j\,|\,X_s=i\,\big)\ , $$ for all $\ s\ $ and $\ t>0\ $, independent of $\ s\ $, which means that the Markov chain is time-homogeneous: \begin{align} \mathbb{P}\big(X_b=j\,\big|\,X_a=i\big)&=\mathbb{P}\big(X_{(b-a)+a}=j\,\big|\,X_a=i\big)\\ &=\mathbf{P}_{ij}(b-a)\ . \end{align} You should be able to use this identity to calculate all the probabilities needed to answer part (b).

• For (c), by the Markov property, for any $\ s\in[0,6)\ $we have \begin{align} \mathbb{P}\big(X_6=1\,\big|&\,X_t=2\ \text{ for all }\ t\in[0,6)\big)\\ &=\mathbb{P}\big(X_6=1\,\big|\,X_t=2\ \text{ for all }\ t\in[s,6)\big)\\ &=\frac{\mathbb{P}\big(X_6=1\ \text{ and }\ X_t=2\ \text{ for all }\ t\in[s,6)\big)}{\mathbb{P}\big(X_t=2\ \text{ for all }\ t\in[s,6)\big)}\\ &\le\frac{\mathbb{P}\big(X_6=1, X_s=2\big)}{\mathbb{P}\big(X_s=2\big)e^{q_{22}(6-s)}}\\ &=\frac{\mathbb{P}\big(X_6=1\,\big|\,X_s=2\big)}{e^{q_{22}(6-s)}}\\ &=\frac{\mathbf{P}_{21}(6-s)}{e^{q_{22}(6-s)}} \end{align} Since this final expression tends to $\ 0\ $ as $\ s\uparrow6\ $, what does that tell you about the value of $\ \mathbb{P}\big(X_6=1\,\big|\,X_t=2\ \text{ for all }\ t\in[0,6)\big)\ $?

• You're given all the entries of $\ \mathbf{P}(t)\ $ explicitly as functions of $\ t\ $, so you should be able to determine whether $\ \lim_\limits{t\rightarrow\infty}\mathbf{P}(t)\ $ exists, and what its value is if it does, simply by doing this for each of the entries of $\ \mathbf{P}(t)\ $ .