Solve a problem using Markov chains

Question

We have the following problem:

At the beginning of every year, a gardener classifies his soil based on its quality: it's either good, mediocre or bad. Assume that the classification of the soil has a stochastic nature which only depends on last year's classification and never improves. We have the following information:

• If the soil is good, then the classification next year will be good ($p=0.2$), mediocre ($p=0.5$) or bad ($p=0.3$).
• If the soil is mediocre, then the classification next year will be mediocre ($p=0.5$) or bad ($p=0.5$).
• If the soil is bad, then the classification next year will be bad ($p=1$).

The gardener can also pay to have the soil fertilized, which will change the probabilities as follows:

• If the soil is good, then the classification next year will be good ($p=0.3$), mediocre ($p=0.6$) or bad ($p=0.1$).
• If the soil is mediocre, then the classification next year will be good ($p=0.2$), mediocre ($p=0.6$) or bad ($p=0.2$).
• If the soil is bad, then the classification next year will be good ($p=0.1$), mediocre ($p=0.4$) or bad ($p=0.5$).

The costs for fertilization are $\$1000$ for good soil, $\$2000$ for mediocre soil and $\$10,000$ for bad soil.

The gardener can sell his soil: good soil sells for $\$10000$, mediocre soil sells for $\$6000$ and bad soil sells for $\$2000$.

When should the gardener fertilize, and when shouldn't he?

I want to tackle this problem using Markov chains but I'm having trouble. I know the state space is given by $S = \{ \mbox{good, mediocre, bad} \}$, but I'm having trouble formulating the Markov chains. Would appreciate help.

Answer 1

Soil condition at start of the year is known, and fertilization will only change the soil next year, so in taking a decision, we should look at its effect on next year's profits.

If soil is "good", next year's profits (thousands of dollars) will be:

$0.2*10 + 0.5*6 + 0.3*2 =5.6$ without fertilization,

and $0.3*10 + 0.6*6 + 0.1*2 - 1 = 5.8 $ with fertilization,

so fertilize

Work out similarly for the other 2 states.

---

Note that you will have to take such decisions every year.