A person often finds that she is up to 1 hour late for work. If she is from $1$ to $30$ minutes late, $\$4 $ is deducted from her paycheck; if she is from $31$ to $60$ minutes late for work, $\$8$ is deducted from her paycheck. If she drives to work at her normal speed (which is well under the speed limit), she can arrive in $20$ minutes. However, if she exceeds the speed limit a little here and there on her way to work, she can get there in $10$ minutes, but she runs the risk of getting a speeding ticket. With probability $\frac{1}{8}$ she will get caught speeding and will be fined $\$20$ and delayed $10$ minutes, so that it takes $20$ minutes to reach work. As she leaves home, let s be the time she has to reach work before being late; that is, $s = 10$ means she has $10$ minutes to get to work, and $s=-10$ means she is already $10$ minutes late for work. For simplicity, she considers s to be in one of four intervals: $(20, \infty), (10, 19), ( 10, 9),$ and $( 20, 11).$ The transition probabilities for s tomorrow if she does not speed today are given by
The transition probabilities for s tomorrow if she speeds to work today are given by
Note that there are no transition probabilities for $(20, \infty)$ and $(-10, 9),$ because she will get to work on time and from $1$ to $30$ minutes late, respectively, regardless of whether she speeds. Hence, speeding when in these states would not be a logical choice. Also note that the transition probabilities imply that the later she is for work and the more she has to rush to get there, the more likely she is to leave for work earlier the next day.
She wishes to determine when she should speed and when she should take her time getting to work in order to minimize her (longrun) expected average cost per day.
Formulate this problem as a Markov decision process by identifying the states and decisions and then finding the $C_{ik}$.
Attempt Let $i=0,1$ the states that represent late and on time
$\begin{array}{c|c|c} decision&action& state\\\hline 1 &speed\ given\ that (-10,9)&0\\\hline 2 &don't\ speed\ given\ that (20,\infty)\ &1\\\hline 3 &speed\ given\ that (-20,11)&0\\\hline 4 &don't\ speed\ given\ that (10,19)&1 \end{array}$
Am I correct so far?
Additionaly,
Why there is no transition probabilities for $(-10,9)$ ?
.. $(-10, 9),$ because she will get to work on time and from $1$ to $30$ minutes late..
Why from $1$ to $30$ minutes late? should not be from $1$ to $9$ minutes late?
Can someone help me with this exercise, please?