Transition probability matrix

321 Views Asked by At

In the article here it had this question.

A walker moves on two positions a and b. She begins at a at time 0, and is at a next time as well. Subsequently, if she is at the same position for two consecutive time steps, she changes position with probability 0.8 and remains in the same position with probability 0.2; in all other cases she decides the next position by a flip of a fair coin. (a) Interpret this as a Markov chain on a suitable state space and write down the transition matrix P. (b) Determine the probability that the walker is in position a at time 10.

Let $X_n$ be the position that the walker is in nth time period. Then isn't the transition matrix is :
$$ \begin{matrix} 0.2 & 0.8 \\ 0.1 & 0.5 \\ \end{matrix} $$
The given answer says

The states are four ordered pairs aa, ab, ba, and bb

The given matrix is:
$$ \begin{matrix} 0.2 & 0.8 & 0 & 0\\ 0 & 0 & 0.5 & 0.5 \\ 0.5 & 0.5 & 0 & 0\\ 0 & 0 & 0.8 & 0.2 \\ \end{matrix} $$
How does it has these four states? Why is my answer wrong. Can someone please explain to me what I did wrong there.

2

There are 2 best solutions below

0
On BEST ANSWER

The transition matrix must take into account not just the current position (that is to say, whether she is at $a$ or $b$ right now), but also the position she was at immediately before. This is because in order to know what the probability of switching positions is going to be, we must know both her current position and the position immediately preceding the current position. Therefore, we encode this as the set of all possible pairs of positions: $$(aa, ab, ba, bb).$$ Then, and only then, is the stochastic process a Markov chain, but we emphasize here that this is Markov on the space of pairs of positions, not individual positions, and each pair of positions is a unique state of the chain.

So, if we look at the pairs of positions, we know, for example, that if her last two positions is $aa$, she transitions to $ab$ with probability $0.8$. But she cannot transition from $aa$ to $ba$ nor $bb$, because the next pair of positions must begin with the same letter that the previous pair ended with. So the row corresponding to $aa$ should read $$\begin{bmatrix}0.2 & 0.8 & 0 & 0\end{bmatrix}$$ if we go by the ordering $(aa, ab, ba, bb)$ described above. The rest is straightforward.

0
On

The answer you provided can't obey the condition that being in a state for two consecutive steps requires a different probability from when you are only in a state for one step. To describe this situation, you need to set conditional probabilities which depend on the two previous states. If the last two states were $[\dots aa]$ you transfer to $[\dots aab]$ with probability 0.8 or $[\dots aaa]$ with probability 0.2. If the last two states were $[\dots ab]$ you can transfer to $[\dots abb]$ or $[\dots aba]$ with probability 0.5 each. If the last two states were $[\dots ba]$ you can transfer to $[\dots bab]$ or $[\dots baa]$ with probability 0.5 each. If the last two states were $[\dots bb]$ you transfer to $[\dots bba]$ with probability 0.8 or $[\dots bbb]$ with probability 0.2.