Steady state of Markov Process

200 Views Asked by At

I would like to have acknowledgment about this exercise.

Find, less than a constant, the steady state of the following Markov process $$\begin{bmatrix} w_{k+1}\\ s_{k+1}\\ d_{k+1} \end{bmatrix} = \begin{bmatrix} 1 & 1/4 &0\\ 0 & 3/4 &1/2\\ 0 & 0 & 1/2 \end{bmatrix} \begin{bmatrix} w_{k}\\ s_{k}\\ d_{k} \end{bmatrix}. $$ Here it is my solution: with direct calculations I found that the matrix has 3 distinct eigenvalues $\lambda_1 =1, \lambda_2=3/4$ and $\lambda_3 = 1/2$, whose corresponding eigenvectors are $(1, 0, 0), (1, -1, 0)$ and $(-2, 1, -2)$ respectively. Setting $u_k:= \begin{bmatrix} w_k\\ s_k\\ d_k \end{bmatrix}$, the previous equality becomes $u_{k+1} = A u_k$ which gives, iterating, that $u_k = A^k u_0$. But the matrix $A$ is diagonalizable, so $A^k =S \Lambda^k S^{-1}$, i.e. $$ \begin{bmatrix} 1 & 1/4 &0\\ 0 & 3/4 &1/2\\ 0 & 0 & 1/2 \end{bmatrix} = \begin{bmatrix} 1 & 1 &-2\\ 0 & -1 & 1\\ 0 & 0 & -2 \end{bmatrix}\begin{bmatrix} 1 & 0 &0\\ 0 & (3/4)^k & 0\\ 0 & 0 & (1/2)^k \end{bmatrix}\begin{bmatrix} 1 & 1 &-1/2\\ 0 & -1 &-1/2\\ 0 & 0 & -1/2 \end{bmatrix}. $$ Thus $$u_k = S\Lambda^k S^{-1} u_0 \iff \begin{bmatrix} 1\\ 0\\ 0 \end{bmatrix} (w_0 +s_0 +d_0) +\begin{bmatrix} 1\\ -1\\ 0 \end{bmatrix} (3/4)^k(-s_0 -1/2 d_0) +\begin{bmatrix} -2\\ 1\\ -2 \end{bmatrix}(1/2)^k (-1/2) d_0. $$ The steady state is obtained evaluating the limit for $k\to +\infty$ of $u_k$, which is, in this case, $$\lim_{k\to +\infty} u_k = w_0.$$

Could anyone please tell me if it is all true or I missed something?

Thank you in advance!

1

There are 1 best solutions below

1
On BEST ANSWER

The terminology is a bit unusual, since there is not really a Markov chain here. If you wish to get the limit of $(u_k)$, then what you did is almost correct, except that you get $$ \lim_{k \to + \infty} u_k = (w_0 + s_0 + d_0) [1, 0, 0]. $$

However, if you are really in the framework of Markov chains, you do not need all this. Denote by $M$ a stochastic matrix, and $Y_k$ the vector describing the probabilities to be in each state after $k$ steps. If $M$ is irreducible and aperiodic, then $Y_k = Y M^k$ converges to the stationary probability, i.e. the unique left-eigenvector of $M$ corresponding to the eigenvalue 1.

In your context, everything is written for right-stochastic matrices, and thus the limit of $(u_k)$ is the unique right-eigenvector of the matrix corresponding to the eigenvalue 1, which is indeed $(1,0,0)$. This is consistent with what is written above since $w_0 + s_0 + d_0 = 1$ for probability vectors.

All calculations apart, this is quite clear: from state 3, you eventually go to state 2; and from state 2, you eventually go to state 1, which is absorbing. Therefore, regardless of where you start, you always end up at state 1 and stay there.