I am having trouble understanding the question and I am not sure if I am heading in the right direction. This is how I have the transition probability matrix:
$$s+1<j<S$$
\begin{array}{c|c|c|c} & j & s+1 & S \\ \hline j & 0 & 0 & z_{1}+z_{2} \\ \hline s+1 & 0 & 0 & z_{4} \\ \hline S & z_{6} & z_{3}+z_{7} & z_{5} \end{array}
I will highly appreciate any guidance or hints.
Hints
The stock level $\ \ell\ $ at the beginning of any period is some integer between $\ s+1\ $ and $\ S\ $. This will be the state of your Markov chain, which therefore has $\ S-s\ $ states, $\ \ell\in\{s+1, s+2,\dots, S\}\ $. If the demand during the period is $\ z\ $ and $\ \ell-z\ge s+1\ $, then the state at the beginning of the next period—that is, the stock level then—will be $\ \ell-z\ $. So what's the probability of that happening, given the value of $\ \ell\ $?
On the other hand, if $\ \ell-z\le s\ $, the stock gets replenished and its level at the start of the next period will be $\ S\ $. Given the value of $\ \ell\ $, what's the probability of that happening?