Markov chain transition for $n$ period with absorbing state

101 Views Asked by At

I have a simple model with a Markov transition.

If the current state is $k$, the next period's state is

$k-1$ with probability $p$,

$k+1$ with probability $q$, and

$k$ with probability $1-p-q$.

However, once $k$ becomes negative, it becomes deterministic and stays there forever.

So, the state space is $S=\{-1,0,1,2,\cdots\}$

My question is

If we start from some integer $m>0$, what is the probability that we end up at $t\in S$ after $n$ times of transitions?

So, I'd like to find the probability distribution over $S$ after $n$ period.

1

There are 1 best solutions below

0
On BEST ANSWER

Since you just want to know what happens after $n$ times starting from $m>0$ you just need to care about the states $E = \{m-n, m-n+1, \cdots, m+n-1, m+n\} \cap \{ -1,0,1, \cdots \}$. You can assume that the last state $m+n$ is absorbing since after $n$ transitions the fact you assume it to be absorbing will change nothing. The same can be assumed to the first state $\max\{m-n, -1\}$, because or it is really absorbing on the original chain (case it is $-1$) or it will change nothing (case it is $m-n>-1$). Now you can use a transition matrix on the states that will matter (defining a new chain $Y$). $$ T = \begin{bmatrix} 1 & 0 & 0 & 0 & \cdots & 0 & 0 & 0 & 0\\ p & 1-p-q & q & 0 & \cdots & 0 & 0 & 0 & 0\\ 0 & p & 1-p-q & q & \cdots & 0 & 0 & 0 & 0\\ \vdots & \vdots & \vdots & \vdots & \ddots & \vdots & \vdots & \vdots & \vdots\\ 0 & 0 & 0 & 0 & \cdots & p & 1-p-q & q & 0\\ 0 & 0 & 0 & 0 & \cdots & 0 & p & 1-p-q & q\\ 0 & 0 & 0 & 0 & \cdots & 0 & 0 & 0 & 1\\ \end{bmatrix} $$ with $T= (p_{i,j})$ where $i,j \in E$, $p_{i,j} = \mathbb{P}_Y(\text{go from state }i\text{ to state }j\text{ after }1\text{ iteration})$ and $\mathbb{P}_Y$ denotes probabilities using chain $Y$.

The matrix $T^n = (p_{i,j}^n)$ gives you the probabilities $\mathbb{P}_Y(\text{go from state }i\text{ to state }j\text{ after }n\text{ iterations})$. The probabilities contained in $T^n$ come from the new chain $Y$, but $Y$ was constructed on a way that the $p_{m,j}^n$ are identical to the probabilities of the original chain. Therefore, the distribution of the original chain $X$ starting from $m$ is given by the line of $T^n$ correspondent to state $m$.

The closed formulas for those probabilities are really ugly. But the good thing is that since the distribution at time $n$ is given by the $|E|-n$-th row of $T^n$ then it is just $e_{|E|-n}^TT^n$ where $\{e_i\}_{i=1}^{|E|}$ is the canonical basis of $\mathbb{R}^{|E|}$. With this observation we can compute the distribution in $O(n^2)$, without making the matrix product. That also gives us a closed formula for $p_{m,j}^n$ (just do the math), but kind of irrelevant.

In case you want to see how those closed formulas look like I wrote a python code here. Here is an example with $n=5$ and $m=2$: $$ \begin{align} p_{2, -1}^n = & p^{4} q + p \left(p^{2} \left(2 p q + \left(- p - q + 1\right)^{2}\right) + p^{2} + 2 p \left(p \left(- p - q + 1\right) + p\right) \left(- p - q + 1\right)\right) + \left(2 p^{3} \left(- p - q + 1\right) + p^{2} \left(p \left(- p - q + 1\right) + p\right)\right) \left(- p - q + 1\right)\\ p_{2, 0}^n = & 4 p^{3} q \left(- p - q + 1\right) + p \left(2 p^{2} q \left(- p - q + 1\right) + 2 p \left(p q + \left(- p - q + 1\right)^{2}\right) \left(- p - q + 1\right) + 2 p \left(2 p q + \left(- p - q + 1\right)^{2}\right) \left(- p - q + 1\right)\right) + \left(- p - q + 1\right) \left(p^{2} \left(p q + \left(- p - q + 1\right)^{2}\right) + p^{2} \left(2 p q + \left(- p - q + 1\right)^{2}\right) + 4 p^{2} \left(- p - q + 1\right)^{2}\right)\\ p_{2, 1}^n = & p \left(p^{2} q^{2} + 8 p q \left(- p - q + 1\right)^{2} + \left(2 p q + \left(- p - q + 1\right)^{2}\right)^{2}\right) + q \left(2 p^{2} \left(2 p q + \left(- p - q + 1\right)^{2}\right) + 4 p^{2} \left(- p - q + 1\right)^{2}\right) + \left(4 p^{2} q \left(- p - q + 1\right) + 4 p \left(2 p q + \left(- p - q + 1\right)^{2}\right) \left(- p - q + 1\right)\right) \left(- p - q + 1\right)\\ p_{2, 2}^n = & p \left(4 p q^{2} \left(- p - q + 1\right) + 4 q \left(2 p q + \left(- p - q + 1\right)^{2}\right) \left(- p - q + 1\right)\right) + q \left(4 p^{2} q \left(- p - q + 1\right) + 4 p \left(2 p q + \left(- p - q + 1\right)^{2}\right) \left(- p - q + 1\right)\right) + \left(- p - q + 1\right) \left(2 p^{2} q^{2} + 8 p q \left(- p - q + 1\right)^{2} + \left(2 p q + \left(- p - q + 1\right)^{2}\right)^{2}\right)\\ p_{2, 3}^n = & p \left(2 q^{2} \left(2 p q + \left(- p - q + 1\right)^{2}\right) + 4 q^{2} \left(- p - q + 1\right)^{2}\right) + q \left(2 p^{2} q^{2} + 8 p q \left(- p - q + 1\right)^{2} + \left(2 p q + \left(- p - q + 1\right)^{2}\right)^{2}\right) + \left(4 p q^{2} \left(- p - q + 1\right) + 4 q \left(2 p q + \left(- p - q + 1\right)^{2}\right) \left(- p - q + 1\right)\right) \left(- p - q + 1\right)\\ p_{2, 4}^n = & 4 p q^{3} \left(- p - q + 1\right) + q \left(4 p q^{2} \left(- p - q + 1\right) + 4 q \left(2 p q + \left(- p - q + 1\right)^{2}\right) \left(- p - q + 1\right)\right) + \left(2 q^{2} \left(2 p q + \left(- p - q + 1\right)^{2}\right) + 4 q^{2} \left(- p - q + 1\right)^{2}\right) \left(- p - q + 1\right)\\ p_{2, 5}^n = & p q^{4} + 4 q^{3} \left(- p - q + 1\right)^{2} + q \left(2 q^{2} \left(2 p q + \left(- p - q + 1\right)^{2}\right) + 4 q^{2} \left(- p - q + 1\right)^{2}\right)\\ p_{2, 6}^n = & 5 q^{4} \left(- p - q + 1\right)\\ p_{2, 7}^n = & q^{5}\\ \end{align} $$