I have a problem which I "simplified" into a simple Markov chain with a constant transition matrix. The transition matrix is a bidiagonal matrix looking like : \begin{align} P = \left( \begin{array}{ccccc} \alpha & 0 & 0 & 0 & 0\\ 1-\alpha & 2\alpha & 0& 0 & 0\\ 0 & 1-2\alpha & 3\alpha & 0 & 0 \\ 0 & 0 & \ddots & \ddots & 0\\ 0 & 0 & 0 & \max(0,1-(k-1)\alpha) & \min(1,k\alpha) \end{array}\right) \end{align} Now I have an initial state $p_0$ defined as $\{1,0,0,\ldots,0\}$.
My objective is to have the values of $P^np_0$ (so the state after $n$ steps in the chain) or at least an approximation/bound on them.
I thought the bidiagonal form would help but there is unfortunately not much documentation on it.
The eigenvalues of $P$ and its transpose $P^T$ are the same, and $P^T$ is upper-diagonal. The diagonal entries $\min(1,i\alpha)$ are therefore the eigenvalues $\lambda_i$ of $P$. You can write $$P = U \Lambda U^{-1},$$ where $\Lambda = \mathrm{diag}(\lambda_1,...,\lambda_k)$ is the diagonal matrix of eigenvalues, and $U$ is the matrix whose $i^{th}$ column is the eigenvector corresponding to $\lambda_i$. Then $$P^n = U \Lambda^n U^{-1}.$$ It remains only to compute $U$, which I leave to you. Your answer will be left-multiplication by the row vector $p_0^T$ (you have written it as right-multiplication, but called it the "initial state" -- one of these is wrong).