Expected number of widgets(Markov chain)

Question

Suppose that a machine making widgets operate at $5$ levels of efficiency ($1$ being the best, $5$ being completely broken). Suppose that the efficiency of the machine from one day to the next evolves according to a Markov chain with the following transition matrix.

$$\begin{pmatrix} 0.9 & 0.05 &0.03 &0.02 & 0 \\ 0&0.9 &0.05 &0.03 &0.02 \\ 0&0 &0.8 &0.1 &0.1 \\ 0&0 &0 &0.6 &0.4 \\ 0& 0 &0 &0 &1 \end{pmatrix}$$

Suppose also that the number of widgets the machine produces in a given day depends on the efficiency level of the machine according to the following table:

\begin{array}{c|c} \text{Efficiency Level} & \text{Widgets Produced}\\\hline 1 & 1000\\ 2 & 900\\ 3 & 700\\ 4 & 500 \end{array}

If a brand new machine is currently operating at efficiency level $1$, what is the expected number of widgets it will be able to produce before the machine completely breaks (efficiency level 5).

What I am doing : Level $5$ is the absorbing state. So, my Q matrix is:

$$\begin{pmatrix} 0.9 & 0.05 &0.03 &0.02 \\ 0&0.9 &0.05 &0.03 \\ 0&0 &0.8 &0.1 \\ 0&0 &0 &0.6 \\ \end{pmatrix}$$

I calculate my R matrix which is: $(I_{4} - Q)$ inverse where I is the identity matrix and it gives me:

$$\begin{pmatrix} 10 & 5 &2.75 &1.5625 \\ 0& 10 &2.5 & 1.375 \\ 0&0 &5 &1.25 \\ 0&0 &0 &2.5 \\ \end{pmatrix}$$

which means that it takes 10 days for the machine to leave level 1 and 5 for level 2 and so on. So, will my final answer be: 17206.25 ( 10*1000 + 5*900 + 2.75*700 + 500*1.5625) or should it be 10*1000 + 10*900 + 5*700 + 500*2.5 = 23250? I am confused about this :(

Answer 1

I wrote this $\texttt R$ code to simulate the Markov chain:

rm(list=ls())

N <- 10000

next_state <- function(i) {
  u <- runif(1)
  if(i==1) {
    if(u < 0.9)
      return(1)
    else if(u < 0.95)
      return(2)
    else if(u < 0.98)
      return(3)
    else
      return(4)
  }
  else if(i==2) {
    if(u<0.9)
      return(2)
    else if(u<0.95)
      return(3)
    else if(u<0.98)
      return(4)
    else
      return(5)
  }
  else if(i==3) {
    if(u<0.8)
      return(3)
    else if(u<0.9)
      return(4)
    else
      return(5)
  }
  else if(i==4) {
    if(u<0.6)
      return(4)
    else
      return(5)
  }
  else
    return(5)
}

widgets <- function(i) {
  if(i==1)
    return(1000)
  else if(i==2)
    return(900)
  else if(i==3)
    return(700)
  else if(i==4)
    return(500)
  else
    return(0)
}

W <- rep(0, N)

for(i in 1:N) {
  state <- 1
  while(state<5) {
    W[i] <- W[i] + widgets(state)
    state <- next_state(state)
  }
}

print(mean(W))

It's giving an output of $\sim17200$-$17300$, for what that's worth.