Hello i have this Markov chain exercise:
Basically we can always move up 1 step, but there is always a possibility that we will go down to the first state 0, the Markov chain consists of N states. When we get to the top we can also stay there with probability p, or go to 0 with probability q(so there are supposed to be 2 p's in the last column)..
The matrix is given by(I have just chosen an N, the N is not fixed but a variable::
\begin{Bmatrix} q & p & 0 & 0 & 0& 0 & 0 & 0 \\ q & 0 & p & 0 & 0& 0 & 0 & 0 \\ q & 0 & 0 & p & 0& 0 & 0 & 0 \\ q & 0 & 0 & 0 & p& 0 & 0 & 0 \\ q & 0 & 0 & 0 & 0& p & 0 & 0 \\ q & 0 & 0 & 0 & 0& 0 & p & 0 \\ q & 0 & 0 & 0 & 0& 0 & 0 & p \\ q & 0 & 0 & 0 & 0& 0 & 0 & p \\ \end{Bmatrix}
I have two questions I am not a able to solve.
1.I am supposed to show that starting in state 0, we will eventually get to state n(I assume that p is bigger than 0). There is a hint to make the last state an absorbing state.
I am not really sure how to solve this. I can maybe solve it with just arguing without any equations like this:? If state N is an absorbing state, then it is also recurrent. The other states are transient. This means that they will only be visited a finite number of times, and hence we have to get to state N at some time. Is this a correct argument?
2. I am supposed to set up equations to that if starting in state i, what is the mean time until I get to state N. And solve these equations, I am able to set them up, but how do I solve them?
Here are the equations: Let $v_i$ be the expected time to reaching state N if we are in state i.
for $i=0,...,N-1$
$v_i=1+q*v_0+p*v_{i+1}$
This equation holds also holds for $v_0$, in order for it to hold for $v_{N-1}$ we note that $v_N=0$.
But I have no idea on how to solve this.
Can someone please help me?
(1) The markov chain is irreducible and finite, therefore it is positive recurrent. therefore it will reach state $N$ eventually, when it is started at state $0$.
(2) The equation you have is correct. So,
$v_{n-1} = 1 + qv_0 + pv_n = qv_0 + 1 $
$v_{n-2} = 1 + q v_0 + pv_{n-1} = (p+1)qv_0 + p + 1 $
$v_{n-3} = 1 + qv_0 + pv_{n-2} = (p(p+1)+1)qv_0 + p(p+1)+1$
So, in general $v_{n-i} = f_{i-1}(p)qv_0 + f_{i-1}(p)$ where $f_i(p) = \sum _{j = 0} ^{i} p^{i} $
We also have $(1-q)v_0 = 1 + pv_1 = 1 + p(f_{n-2}(p)qv_0 + f_{n-2}(p)) $. So, we can solve for $v_0$.
There is also an easier way to solve, which you can think about.