Suppose that for all $\alpha \in (0,1)$, the matrix $\mathbf I-\alpha \mathbf A$ is invertible, but that $\det(\mathbf I-\mathbf A)=0$. How do I calculate
$$ \lim_{\alpha \to 1} (\mathbf I-\alpha \mathbf A)^{-1}(\mathbf I- \mathbf A)?$$
Suppose that for all $\alpha \in (0,1)$, the matrix $\mathbf I-\alpha \mathbf A$ is invertible, but that $\det(\mathbf I-\mathbf A)=0$. How do I calculate
$$ \lim_{\alpha \to 1} (\mathbf I-\alpha \mathbf A)^{-1}(\mathbf I- \mathbf A)?$$
On
In the special case where $A_{n \times n}$ is a real symmetric matrix the limit can be explicitly computed in terms of the spectral decomposition of $A$.
The condition $\det(I-A)=0$ ensures $1$ is an eigenvalue of $A$.
The condition $\det(I-\alpha A) \neq 0$ for $\alpha \in (0,1)$ implies $\det(\frac{1}{\alpha}I -A) \neq 0$ for $\alpha \in (0,1)$, i.e., $\det(\mu I-A) \neq 0$ for $\mu > 1$. So all eigenvalues of $A$ that are distinct from $1$ (if any) must lie in the interval $(-\infty,1)$.
Let $r > 0$ be the multiplicity of $1$ as an eigenvalue of $A$, and let $\lambda_{r+1},\dots,\lambda_{n}$ denote the remaining eigenvalues, with each eigenvalue repeated as many times as its multiplicity.
We can find an orthronormal basis of $\mathbb{R}^n : \{u_1,u_2,\dots,u_r,u_{r+1},\dots,u_n\}$ such that $$\label{e:1}A = \sum_{i=1}^{r}u_i u_i^T + \sum_{i=r+1}^{n}\lambda_iu_iu_i^T \tag{*}$$ and the orthonormality implies $$ \label{e:2} I = \sum_{i=1}^{n}u_iu_i^T \tag{+}. $$
We have from $\eqref{e:1}$ and $\eqref{e:2}$ $$I-A = \sum_{i=r+1}^{n}(1-\lambda_i)u_iu_i^T$$ and $$ I - \alpha A = \sum_{i=1}^r (1 - \alpha)u_iu_i^T + \sum_{i=r+1}^{n}(1-\alpha\lambda_i)u_iu_i^T. $$
Note for $\alpha \in (0,1)$ and $i > r$ we have $1 -\alpha \lambda_i \neq 0$ so $$ (I-\alpha A)^{-1} = \frac{1}{1-\alpha}\sum_{i=1}^ru_i u_i^T + \sum_{i=r+1}^n\frac{1}{1-\alpha\lambda_i} u_iu_i^T $$ and so $$ (I-\alpha A)^{-1}(I - A) = \sum_{i=r+1}^{n}\frac{1-\lambda_i}{1-\alpha\lambda_i}u_iu_i^T. $$ So, $$ \lim_{\alpha \to 1^{-}}(I-\alpha A)^{-1}(I - A) = \sum_{i=r+1}^{n}u_iu_i^T. $$
If everything were as nice as it could possibly be, we could carry out the following calculation:
$$\lim_{\alpha \to 1^-} (I-\alpha A)^{-1}(I-A) \\ =\lim_{\alpha \to 1^-} \sum_{n=0}^\infty \alpha^n (A^n-A^{n+1}) \\=\lim_{N \to \infty} \sum_{n=0}^N A^n-A^{n+1} \\=\lim_{N \to \infty} I-A^{N+1} \\= I-\lim_{N \to \infty} A^N.$$
Assume that $A$ has no eigenvalues of modulus $\geq 1$ except $1$, and the eigenvalue $1$ of $A$ (which you have already assumed exists) is non-defective. Then $\lim_{N \to \infty} A^N$ is a projection onto the eigenspace of $A$ with the eigenvalue $1$. In general it may not be orthogonal in the usual inner product, so to uniquely specify it we should give its coprojector. This coprojector is the projection onto the span of the other (generalized) eigenvectors of $A$.
If either of these assumptions are violated then we should not expect $\lim_{N \to \infty} A^N$ to exist (either by oscillation or by blowup).
The one sketchy step here is the second equality, which required an interchange of limits that I didn't justify. These sorts of "critical" situations can cause such interchanges to break down, so you should check this step carefully.