Steady-state probability for a Markov process

528 Views Asked by At

enter image description here

Friends,

I got stuck with formulating a Markov chain that I just came up with.

What I want to do is to obtain a steady-state expression for $b_0$ as a function of $b_k$, in order to plug in the normalization condition: \begin{align} \sum_{k=0}^{CW-1} b_k = 1 \end{align} so I can have an explicit expression for $b_0$ in terms of CW and $\mathsf{P}_b$.

One constraint is: if not reached $b_0$ within $M$ transitions, the process fails and start over from the top with $1/CW$.

Any thoughts will be greatly appreciated!!

1

There are 1 best solutions below

0
On

First consider the chain where you identify the $b_n$ and $D_n$ states for $n \geq 1$. Say the top state is called $s_0$, then you have $s_0 \to s_n,n=1,\dots,N$ with probability $1/N$, $s_1 \to s_0$ with probability $1$, $s_n \to s_{n-1}$ with probability $1-P_b$ for $n \geq 2$, and $s_n \to s_n$ with probability $P_b$ for $n \geq 2$. (Watch out that I have relabeled things compared to your notation.)

Now you read the equations for the steady state distribution:

$$\pi_0=\pi_1$$

$$\pi_1=(1-P_b)\pi_2+\frac{1}{N} \pi_0 \Rightarrow \pi_2 = \frac{1-1/N}{1-P_b} \pi_0$$

$$\pi_n=P_b \pi_n + (1-P_b) \pi_{n+1}+\frac{1}{N} \pi_0 \Rightarrow \pi_{n+1}=\pi_n-\frac{1}{N(1-P_b)} \pi_0 \quad 2 \leq n<N$$

$$\pi_N=P_b \pi_N + \frac{1}{N} \pi_0 \Rightarrow \pi_N=\frac{1}{N(1-P_b)} \pi_0$$

Thus what you see is that $\pi_n$ are just dropping by $\frac{1}{N(1-P_b)} \pi_0$ as you sweep across (which I first observed numerically, by the way). So basically there is some $C$ such that $\pi_0=C,\pi_1=C,\pi_n=\frac{N-n+1}{N} C$ for $n \geq 2$, and then you can find $C$ by using the normalization requirement.

Now you just have to compute the expected number of visits to each of $b_n$ and $D_n$ before actually leaving that pair of states. This is easily done using the geometric distribution: one visit to the pair will visit $b_n$ once and it will visit $D_n$ a Geo($1-P_b$) number of times, with the convention that the geometric random variable can be zero. Such a thing has mean $\frac{P_b}{1-P_b}$, which is the important quantity. So the fraction of $\pi_n$ for $n \geq 2$ that should be assigned to $b_n$ is $\frac{1}{1+\frac{P_b}{1-P_b}}=1-P_b$ and thus the fraction that should get assigned to $D_n$ is $P_b$. (Note that we could have removed the self-loops entirely from the "consolidated chain", handling all of them at this stage instead.)

(This does not take your fail condition into account, which makes the chain non-homogeneous and thus makes the calculation much more difficult.)

By the way, in this answer I use this concept of aggregating several states together into one state, defining an aggregated chain, solving for some property of the aggregated chain, and then resolving the internal dynamics of the aggregate at the end. This isn't specific to this situation; a much more advanced subject that uses this same idea is the notion of Freidlin cycles, a component of Freidlin-Wentzell theory, which is a branch of large deviation theory related to random perturbations of dynamical systems.