Let $\|.\|_2$ denote the matrix norm induced from the Euclidian vectornorm and let
\begin{align} M=\left(\begin{array}{cc} a+b & -b \\ 1 & 0 \end{array}\right). \end{align} I need to bound $\|M^n\|_2$ for any $n\in \mathbb{N}$. It is assumed that $a\in[0,1]$ and $b\in(0,1)$. I tried to use Gelfand's formula from which we obtain: $$\|M^n\|_2\leq (\rho(M)+o(1))^n,$$ where $\rho(M)$ denotes the spectral radius of $M$. It is easy to see why $\rho(M)\leq \max\left\{1-\frac{1-a}{2},b\right\}$. Therefore $\|M^n\|_2<c$ for any n, if $\, a\leq q<1$. In the case $a=1$ we see that: \begin{align*} \|M^n\|_2=\left\|\frac{1}{b-1}\left(\begin{array}{cc} b^{n+1}-1 & -b-b^{n+1} \\ b^{n}-1 & -b-b^{n} \end{array}\right)\right\|_2\leq c. \end{align*} But I have no idea how to bound $\|M^n\|_2$ when $a \rightarrow 1$, since I haven't found any results on the convergence of Gelfand's formula. Any suggestions how to prove this? Also according to many plots, it should hold that
\begin{align*}
\|M^n\|_2\leq\left\|\frac{1}{b-1}\left(\begin{array}{cc}
b^{n+1}-1 & -b-b^{n+1} \\
b^{n}-1 & -b-b^{n}
\end{array}\right)\right\|_2\leq c,
\end{align*}
for any $a\in[0,1]$. Any idea how to prove the last inequality?
I show that there is a constant $K>0$ such that for all $a\in[0,1]$, $b\in]0,1[$ and all positive integers $n$ we have $$\tag{1}||M^n||_2\leq\frac K{1-b} \mbox{ for }M=\begin{pmatrix}a+b&-b\\1&0\end{pmatrix}.$$ As a numerical value for $K$, we obtain $K=2$.
Observe that there cannot exist a constant $L$ such that $||M^n||_2\leq L$ for all the above $a,b,n$. If such an $L$ existed we would have $||M^n||_2\leq L$ also for $a=b=1$ by continuity, that is for $M=\begin{pmatrix}2&-1\\1&0\end{pmatrix}.$ Now this matrix has a double eigenvalue $\lambda=1$ and $M=I+N$ with the nilpotent matrix $N=\begin{pmatrix}1&-1\\1&-1\end{pmatrix}.$ Hence $M^n=I+nN$ for all $n$ and $||M^n||_2$ is unbounded.
It would be simpler to show for each $(a,b)$ that either both eigenvalues of $M$ are inside the open unit disk or that both are inside the closed unit disk and different and then to conclude boundedness of $||M^n||_2$ in each case. The eigenvalues are also discussed in the proof below. The aim of the proof is to find the best possible type of bound valid for all $a,b$ considered in the question.
Proof of (1): The characteristic polynomial of $M$ is $p(x)=x^2-(a+b)x+b$. Therefore $M^2=(a+b)M-bI$ and hence $M^{n+1}=(a+b)M^n-bM^{n-1}$ for all $n$. Hence there exist sequences $c_n,d_n$ such that $M^n=c_n M+d_nI$ for all $n$ and $c_{n+1}=(a+b)c_n-bc_{n-1}$, $d_{n+1}=(a+b)d_n-bd_{n-1}$. We also have $M^{n+1}=M\,M^n=((a+b)c_n+d_n)M-bc_nI$ and hence $d_n=-bc_{n-1}$. We have shown that $$\tag{2}M^n=c_nM-bc_{n-1}I \mbox{ where } c_{n+1}=(a+b)c_n-bc_{n-1} \mbox{ and }c_0=0,c_1=1.$$
So it suffices to show that there is a constant $K$ such that $|c_n|\leq K/(1-b)$ for all $a\in[0,1]$, $b\in]0,1[$ and all positive integers $n$.
The sequences $\{x_n\}_n$ satisfying $$\tag3 x_{n+1}=(a+b)x_n-bx_n$$ form a vector space of dimension two. If $\lambda_1,\lambda_2$ are the roots of $p(x)$, i.e. $\lambda_j^2=(a+b)\lambda_j-b$ then we can give a basis of this vector space. If the roots are different, then $\{\lambda_j^n\}_n$, $j=1,2$ form a basis; if there is a double root then $\{\lambda_1^n\}_n$ and $\{n\lambda_1^n\}_n$ form a basis.$\newcommand{\l}{\lambda}$
Case 1: $p$ has a double root $\l$. Then $\l=\sqrt b$ and $a+b=2\l=2\sqrt b$. Hence $a=2\sqrt b-b\in]0,1[$ because $0<b<1$. Using the initial conditions, we find that $c_n=n\l^{n-1}$ for all $n$. We show $$\tag4 n\l^{n-1}\leq\frac1{1-\l}\mbox{ for all positive }n\mbox{ and }\l\in[0,1[.$$ Indeed, the function $f(\l)=n\l^{n-1}(1-\l)$ assumes its maximum in $\l=\frac{n-1}n$ and this maximum is $\left(\frac{n-1}n\right)^{n-1}$. As $$\frac1{1-\sqrt b}=\frac{1+\sqrt b}{1-b}\leq \frac2{1-b}$$ (2) and (4) prove the statement in this case.
Case 2: $\l_1\neq\l_2$. Here we find from the initial conditions that $$\tag5 c_n=\frac{\l_1^n-\l_2^n}{\l_1-\l_2}=\l_1^{n-1}+\l_1^{n-2}\l_2+\cdots+\l_2^{n-1}.$$ If both roots are complex conjugates, then we have $|\l_1|=|\l_2|=\sqrt b$ because $|\l_1|^2=\l_1\overline{\l_1}=\l_1\l_2=b$. Hence $|c_n|\leq n\sqrt b^{n-1}$ and we conclude as in case 1 that $|c_n|\leq\frac2{1-b}$.
It remains to treat the case of two different real roots. As $\l_1+\l_2=a+b>0$ and $\l_1\l_2=b>0$, both are positive. Since $p(1)=1-a\geq0$ and $p'(x)=2x-a-b>0$ for all $x\geq1$ both roots are in $]0,1]$. We assume $0<\l_1<\l_2\leq1$. Observe that $\l_2=1$ if $a=1$.
We also have $$\l_1=\frac{a+b}2-\sqrt{\left(\frac{a+b}2\right)^2-b}$$ and hence $1-\l_1\geq1-\frac{a+b}2\geq\frac{1-b}2$. Since their sum equals $1-\l_1$, we have $\l_2-\l_1\geq\frac{1-b}4$ or $1-\l_2\geq\frac{1-b}4$.
In the first case, $|c_n|\leq\frac1{\lambda_2-\l_1}\leq\frac4{1-b}$.
In the second case, we have $|c_n|\leq n\l_2^{n-1}$ and using (4), we obtain $|c_n|\leq \frac 4{1-b}$. This completes the proof.
Remarks: In all the cases, we find that $|c_n|\leq\frac4{1-b}$ for all $n$ and hence by (2), we find that (1) holds with $K=4(\sup_{a,b}||M||_2+1)=4(2+\sqrt2)\approx13.66$.
The estimates of the above proof can actually be sharpened. At the same time, the second question will be answered. Observe first that (2) also implies that $$\tag6 M^n=\begin {pmatrix}c_{n+1}&-bc_{n}\\ c_n&-bc_{n-1}\end{pmatrix}.$$ This can also be proved by induction using the recursion for the sequence $\{c_n\}_n$. In the special case $a=1$, (5) implies $$M^n=\frac1{1-b}\begin {pmatrix}1-b^{n+1}&-b+b^{n+1}\\ 1-b^n&-b+b^n\end{pmatrix}.$$ Except for two signs, this is a formula mentioned in the question.
Later we show that $$\tag7|c_n|\leq\frac{1-b^n}{1-b}\mbox{ for all }a\in[0,1],b\in]0,1[\mbox{ and nonnegative integers }n. $$ Now the definition of the $\|.\|_2$-norm for matrices shows that $$\tag8\left\|\begin {pmatrix}u_{1}&u_{2}\\ u_3&u_{4}\end{pmatrix}\right\|_2\leq \left\|\begin {pmatrix}v_{1}&v_{2}\\ v_3&v_{4}\end{pmatrix}\right\|_2$$ if $|u_i|\leq v_i$ for $i=1,2,3,$. Hence (6) and (7) imply that $$\|M^n\|_2\leq \left\|\frac1{1-b}\begin {pmatrix}1-b^{n+1}&b-b^{n+1}\\ 1-b^n&b-b^n\end{pmatrix}\right\|_2.$$ This answers the second question and corrects a sign error in it. Furthermore since $1-b^n\leq1$ etc, we obtain with (8) $$\|M^n\|_2\leq\frac1{1-b}\left\|\begin {pmatrix}{1}&{1}\\ 1&{1}\end{pmatrix}\right\|_2\leq\frac2{1-b}.$$
It remains to prove (7). Here we use formula (5),the second part of which is valid also if $x^2-(a+b)x+b$ has a double root.
Case 1: $x^2-(a+b)x+b$ has two complex comjugate roots $\l_1,\l_2$ or a double root $\l_1=\l_2$. Using $|\l_1|=|\l_2|=\sqrt b$, we find that $$|c_n|\leq n\,b^{(n-1)/2}\leq 1+b+\cdots + b^{n-1}=\frac{1-b^n}{1-b}.$$ Here we used that $2\,b^{(n-1)/2}\leq b^i+b^{n-1-i}$ for $0\leq i<\frac12(n-1)$ and added all these inequalities and a single middle term in the case of odd $n$.
Case 2: $x^2-(a+b)x+b$ has two different real roots $\l_1,\l_2$. Recall that $b=\l_1\l_2$ and that we can assume that $0<\l_1<\l_2\leq1$. For convenience, we treat only even $n=2m$, the modifications for odd $n$ are left to the reader. Using (5), it is sufficient to show that $$\tag9 \l_2^{n-1}+b\l_2^{n-3}+\cdots+b^{n-2}\l_2^{-n+3}+b^{n-1}\l_2^{-n+1}\leq1+b+\cdots+b^{n-2}+b^{n-1}$$ if $\sqrt b<\l_2\leq1$. This in turn follows from the fact that for any nonnegative integer $k\leq m-1$, the function $g:]\sqrt b,1]\to\mathbb R$, $g(x)=b^{k}x^{n-1-2k}+b^{n-1-k}x^{2k-n+1}$, has a positive derivative. Therefore it is strictly increasing and hence $g(\l_2)\leq g(1)$ for $\l_2\in]\sqrt b,1]$.