Starting from state 0, compute the mean number of visits of state 1 from one-step transition probability matrix.

698 Views Asked by At

A Markov chain has one-step transition probability matrix $$ \mathrm P= \begin{pmatrix} 1/4 & 1/4 & 1/4 & 1/4 \\ 1/4 & 1/4 & 1/2 & 0 \\ 0 & 0 & 1/2 & 1/2 \\ 0 & 0 & 1/3 & 2/3 \\ \end{pmatrix} $$

Starting from state 0, compute the mean number of visits of state 1.

First, I set up 3 equations. $$m_0=1+\frac 14(m_0+m_1+m_2+m_3)$$ $$m_2=1+\frac 12(m_2+m_3)$$ $$m_3=1+\frac 13m_2+\frac 23m_3$$

Second, set $m_1=0$ then solve the above equations, I find the following one by gaussian elimination.

$$ \begin{pmatrix} 3 & -1 & -1 | 4 \\ 0 & 1 & -1 | 2 \\ 0 & 1 & -1 | -3 \\ \end{pmatrix} $$

Am i wrong in some steps?

1

There are 1 best solutions below

0
On BEST ANSWER

Let $$\tau = \inf\{n>0: X_n\in\{2,3\}\}$$ the first passage time into $\{2,3\}$. Let $Y$ be the number of visits to state $1$. Then for each $n>0$, $Y\mid \tau=n$ follows a $\operatorname{Bin}\left(n-1,\frac12\right)$ distribution. Since $\tau$ itself has $\operatorname{Geo}\left(\frac12\right)$ distribution, we compute \begin{align} \mathbb E[Y] &= \sum_{n=1}^\infty\mathbb E\left[Y\mid \tau=n\right]\mathbb P(\tau = n)\\ &= \sum_{n=1}^\infty \frac12(n-1)\left(\frac12\right)^n\\ &= \frac14\sum_{n=0}^\infty (n+1)\left(\frac12\right)^n\\ &= \frac14\left(\frac1{\left(1-\frac12\right)^2} \right)\\ &= 1. \end{align}