Probability of being at given state in a continuous time Markov chain

Question

Question:

Consider a two-state continuous time Markov chain (with states $1$ and $2$) in which the holding rate at state $1$ is $\lambda_1=2$, and the holding rate at state $2$ is $\lambda_2=3$.

Suppose that we start at state $1$ (i.e. $X_0 = 1$). Find the probability that $X_t = 1$ as $t \rightarrow \infty$.

---

Attempt:

So the transition matrix for the underlying discrete Markov chain is

$$\mathbf P = \begin{pmatrix} 0 & 1 \\ 1 & 0 \end{pmatrix}$$

whereas the $Q$-matrix for the continuous time Markov chain is

$$\mathbf Q = \begin{pmatrix} -2 & 2 \\ 3 & -3 \end{pmatrix}$$

Let $\xi = (\xi_1 , \xi_2)$ be the stationary distribution. At this point, do I solve $\xi \mathbf P = \xi$ or $\xi \mathbf Q = 0$ to find the stationary distribution?

And, after I have found the stationary distribution, is the required probability simply $\xi_1$?

Any hints/suggestions would be much appreciated. Thanks!

Answer 1

You seem to want to use row vectors where most people work with column vectors for the distribution vector. In terms of the distribution $\vec{\xi}(t) = \pmatrix{\xi_1(t)\\ \xi_2(t)}$ the matrix describing the evolution of the system is $$ \frac{d\vec{\xi}}{dt} = \pmatrix{-2 & 3\\ 2 & -3} \vec{\xi} $$ You are trying for a stationary state, in which $ \frac{d\vec{\xi}}{dt} = 0$ so you want to solve the simultaneous equations $$ \pmatrix{-2 & 3\\ 2 & -3} \pmatrix{\xi_1\\ \xi_2} = 0 \\ \xi_1+\xi_2 = 1 $$ and you answer for the probability of state $1$ is indeed $\xi_1$ per that solution.

Note that although there appear to be $3$ equations in $2$ unknowns, the first two equations are not independant so there will be a solution.

Answer 2

First we can do some reasoning about why we use $\pi Q = 0$ to get the stationary distribution. According to the definition of stationary distribution, the equation we are to solve is $\pi P(t)=\pi$, where $P(t)$ is the transition matrix of the process.

One thing to note is that the $\mathbf{P}$ you mentioned is not the same as $P(t)$, where the latter is a function of $t$. $\mathbf{P}$ you mentioned is the jump matrix. It is kind of like what you said, an underlying discrete-time transition matrix.

Let's get back to the equation $\pi P(t)=\pi$. Obviously it is not easy to solve. Hence we can take derivative with respect to $t$, and the LHS becomes $\pi Q$, and the RHS becomes $0$.

To answer your second question. The question asked explicitly for $\mathbb{P}(X(t)=1)$ as $t\rightarrow \infty$. Hence the distribution we want is actually the limiting distribution. Under the condition that $X$ is irreducible with a standard semigroup $\{\mathbb{P}(t),t\geq 0\}$ of transition probabilities, we can say that the stationary distribution is also the limiting distribution.

Also, for this question the full balance equation $\pi Q = 0$ is also the detailed balance equation, indicating the process is reversible.