How do I construct transition matrix for the following?

Question

A shopkeeper runs his shop in an area that typically gets heavy rains. He has three umbrellas.

Every day, he goes to his shop in the morning and comes back home in the evening. If it is raining in the morning, he would carry an umbrella on the way to the shop (unless if, unfortunately, all three umbrellas happen to be at the shop). Likewise, if it is raining in the evening, he would carry an umbrella on his way back home (unless if, unfortunately, all three umbrellas happen to be at his home). If it is raining and he doesn’t have an umbrella, he would still need to go to his shop (or come back home) and will unfortunately end up getting wet.

If it is not raining in the morning, and all three umbrellas are at home, he would nevertheless carry an umbrella to the shop just in case it rains in the evening. However, if he has goods to carry to his shop that day, he would not be able to carry an umbrella and would run the risk of getting wet in the evening. Likewise, if it is not raining in the evening, and all three umbrellas are at shop, he would nevertheless carry an umbrella back home just in case it rains next morning. However, if he has goods to carry back home that evening; he would not be able to carry an umbrella and would run the risk of getting wet the next day.

In other words, he would always carry an umbrella if:

1. It is raining and he has an umbrella to carry from his starting place; or
2. It is not raining, he has all three umbrellas at his starting place and he does not have any goods to carry.

The probability that he has goods to carry from one place to the other (home to shop or shop to home) is 75%.

The probability that it’s raining in the morning is p. Likewise, the probability that it’s raining in the evening is also p. Rains in the morning and in the evening are events independent of each other.

I want to construct a Markov chain and write a transition matrix for it. I am getting confused while writing the possibilities.

Can someone help me how to approach this problem step-by-step, if possible with the help of a probability tree/in table format?

The transition matrix is as follows:

\begin{bmatrix} 0 & 0 & 0 & 1 \\ 0 & 0 & (1-p) & p \\ 0 &(1-p) &p & 0 \\ 0.75(1-p) &0.25(1+3p) & 0 & 0 \end{bmatrix}

You may define $X_i$ to represent the number of umbrellas at a place where the shopkeeper presently is. $X_i$ can take the values 0, 1, 2 and 3.

Answer 1

The first thing to do is enumerate the number of states. There are four state variables:

• The number of umbrellas at home: $u \in \{0,1,2,3\}$
• Whether its the morning (0) or evening (1): $t \in \{0,1\}$
• Whether its not raining (0) or raining (1): $r \in \{0,1\}$
• Whether he does not (0) or does (1) have goods to carry: $g \in \{0,1\}$

The shopkeeper takes a single action:

• Whether to not carry (0) or carry (1) an umbrella: $a(u,t,r,g) \in \{0,1\}$

The paragraph describes his action function:

• $a(u,0,1,g) = 1$ if $u > 0$
• $a(u,1,1,g) = 1$ if $u < 3$
• $a(3,0,0,0) = 1$
• $a(0,1,0,0) = 1$
• $a(u,t,r,g) = 0$ otherwise

The state transitions are governed by

\begin{align} u_{s+1} &= \begin{cases} u_s - a_s \text{ if $t_s = 0$} \\ u_s + a_s \text{ if $t_s = 1$}\end{cases}\\ t_{s+1} &= \begin{cases} t_s+1 \text{ if $t_s = 0$} \\ t_s-1 \text{ if $t_s = 1$}\end{cases}\\ r_{s+1} &= \begin{cases} 0 \text{ with probability $p$} \\ 1 \text{ with probability $(1-p)$}\end{cases}\\ g_{s+1} &= \begin{cases} 0 \text{ with probability $.25$} \\ 1 \text{ with probability $.75$}\end{cases} \end{align} where $s$ indexes the time periods.

This provides all the information you need to construct the transition matrix. Notice that because the state space has $4 \times 2 \times 2 \times 2 = 32$ points, the transition matrix will be quite large ($32 \times 32$).

Answer 2

I am really reverse engineering your matrix, but that should not matter if you are reading this.

• As you have edited, $X_i$ is the number of umbrellas where the shopkeeper is.
• The convention is $x_n A = x_{n+1}$.
• With no umbrellas $x_n=0$, we are forced to have $x_{n+1} = 3$
• If he has 1 umbrella $x_n=1$, then $x_{n+1} = \begin{cases} 3 & \text{rains} \\ 2 &\text{no rain} \end{cases}$

• If $x_n=2$, then $x_{n+1} = \begin{cases} 2 & \text{rains} \\ 1 &\text{no rain} \end{cases}$

• If $x_n = 3$, only then does the carrying of goods matter: $x_{n+1} = \begin{cases} 0 & \text{there are goods to carry and it does not rain} \\ 1 &\text{no goods to carry, or there are goods to carry and it rains} \end{cases}$

Although technically correct as you only impose the 'goods' restriction in the paragraph where you explain the scenario of all 3 umbrellas being stuck somewhere, I find this unrealistic because it tells me that somehow, only when he has 3 umbrellas, one of the following is magically true:

1. He has goods to carry (and not when he has <3 umbrellas), or
2. He is weakened and can only carry an umbrella xor the goods(and he is perfectly happy to carry both umbrella and goods when he has <3 umbrellas)