Probability of visiting a state for the first time in $n=k$ without knowing initial state.

Question

Let a markov chain with $6$ states. $$ p= \begin{bmatrix} 1 & 0 & 0 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 & 0 & 0 \\ 0 & \frac{1}{4} & \frac{1}{4} & \frac{1}{2} & 0 & 0 \\ 0 & 0 & 0 & \frac{1}{2} & \frac{1}{2} & 0 \\ 0 & 0 & 0 & \frac{1}{3} & \frac{1}{3} & \frac{1}{3} \\ 1 & 0 & 0 & 0 & 0 & 0 \\ \end{bmatrix} $$ At time $n=0$ a random state is picked as an initial state (probability for each state is $\frac{1}{6}$). How should I calculate the probability of the system to visit state $5$ for the first time at $n=k$?

Answer 1

Consider modifying the markov chain so as to have state 5 as an absorbing state... that is to say, if you are in state 5 at a particular time, you remain at state 5.

I see that you write your transition matrix with rows adding to 1 so I will try to match notation... (I personally prefer columns adding to 1).

So then, the probability of being in state 5 at time $k$ would be the fifth entry of $v_0 p^k$ where $v_0$ is your initial state vector (here all entries equal to $\frac{1}{6}$).

The probability of being in state 5 at time $k-1$ would be the fifth entry of $v_0p^{k-1}$.

The probability of having entered state 5 specifically at the first time at time $k$ would then be equal to the probability of being in state 5 at time $k$ minus the probability of being in state 5 before time $k$... thus making it the fifth entry of $v_0(p^k-p^{k-1})$