I have the following exercise and I have some doubts about it's solution.
a) Find $P\{X_4=1\}$.
b) Calculate the limiting distribution.
c) What is the long run rate of repairs per unit time?
My attempt:
For a) I think first I need to establish the transition matrix, but I'm not sure if this way is correct:
$$ \begin{pmatrix} .9 & .1 & 0 \\ 0 & .9 & .1 \\ 1 & 0 & 0 \\ \end{pmatrix} $$ I think it's not correct because they're asking me for $P\{X_4=1\}$ so the matrix needs the fourth entry?
On
I think your problem is that you've simply written down a transition matrix by reflex, but haven't grokked what it means:
$$ \begin{pmatrix} P(X_{n+1} = 1) & P(X_{n+1} = 2) & P(X_{n+1} = 3) \end{pmatrix} = \begin{pmatrix} P(X_{n} = 1) & P(X_{n} = 2) & P(X_{n} = 3) \end{pmatrix} \begin{pmatrix} .9 & .1 & 0 \\ 0 & .9 & .1 \\ 1 & 0 & 0 \\ \end{pmatrix} $$
Where, I assume, $P_{ij}$ means the probability of a machine in state $i$ transitioning to state $j$.
The state space of the given Markov chain has three states $$S=\{1:\text{Good Operating Order, 2:Deteriorated Operating Order, 3:In Repair}\}.$$ $X_{n}$ denotes the state that the Markov chain occupies at time $n$. The probabilities that the chain can be found in different states at time $n$ is specified using a row vector, denoted by $p^{(n)}$. It is also called the state probability vector at time $n$. By definition, then $$p^{(n)}=(P\{X_{n}=1\},P\{X_{n}=2\},P\{X_{n}=3\}).$$ For example, $p^{(2)}=(0.2,0.5,0.3)$ means that the process could be found in state-1 with probability $0.2$, in state-2 with probability $0.5$ and in state-3 with probability $0.3$ at time $n=2$.
To study a Markov chain, what all we require is, a transition probability matrix $P$ , and the initial probability distribution of the chain. The initial probability distribution, by convention, is defined by a row vector. By initial probability distribution of the chain we mean with what probability the chain could start in different states of the chain at time $n=0$. In other words, it is the vector $$(P\{X_{0}=1\},P\{X_{0}=2\},P\{X_{0}=3\})$$
We are given that $X_{0}=1$, which means that the process starts in state-1. This in turn means that the initial probability distribution of the chain is $p^{(0)}=(1,0,0)$. By pre-multiplying the $P$ with $p^{(0)}$, we get the state probability vector at time 1. That is, $p^{(1)}=p^{(0)}P$. Similarly, $p^{(2)}=p^{(1)}P=(p^{(0)}P)P=p^{(0)}P^{(2)}$, where $P^{(2)}$ is the two-step transition probability matrix. In general, $$p^{(n)}=p^{(0)}P^{(n)}.$$ So, to calculate the state probabilities at time $n$, we require the initial probability vector and the n-step transition probability matrix.
To answer (a), compute $P^{(4)}$ and pre-multiply it by $p^{(0)}$ and pick up the first element of that vector, since you are interested in with what probability that the process will be found in state-1.