Markov Process With Variable State

30 Views Asked by At

I have a transition graph in which there exist three states $s_1,s_2,s_3$ such that: if the transition from $s_1$ to $s_3$ happens, then $s_3$ is an absorbing state. Otherwise, if the transition from $s_2$ to $s_3$ happens, then you can transit from $s_3$ to another state with positive probability.

In other words, state $s_3$ is a variable state; sometimes it is absorbing and sometimes it is not. It is absorbing if the previous state was $s_1$ and it can be absorbing or not if the previous state was $s_2$.

Is this still a Markov process? If so what do we call it?

1

There are 1 best solutions below

0
On BEST ANSWER

You have not described a Markov process, because the current state is insufficient to completely determine the transition probabilities to the next state.

Specifically, if I tell you that you are in state $s_3$, you still need to know your previous state (i.e. whether you came from $s_1$ or $s_2$) to determine the conditional probability, $P\big(s(t+1) \ \big| \ s(t)=s_3, s(t-1)\big) \neq P\big(s(t+1) \ \big| \ s(t)=s_3\big)$.

As usual, you can make your process Markovian by enlarging your state space to include the missing information. "Break" $s_3$ into two states (now your state space has cardinality $4$), call them $s_{3a}$ (an absorbing state) and $s_{3b}$ (a state with whatever positive exit probabilities you intended).

Then, only $s_1$ is connected to $s_{3a}$ and only $s_2$ is connected to $s_{3b}$ to achieve behavior identical to what you described. (Note $s_1$ and $s_2$ can still be connected to each other).

To compare, try writing a transition matrix for your proposed process (see that you cannot), and then try writing it for what I have proposed (see that it agrees with your intent).

As for naming, I don't know of any special properties your proposed process satisfies, so I'd just call it a "discrete stochastic process", while with the modification I described, it is a "discrete Markovian stochastic process", or just "Markov chain" for short.