Let $X$ be a Banach space, $A,B:X\to X$ be continuous/bounded linear operators where $I-AB$ and $I-BA$ are compact. I want to show that $\dim(ker(A))<\infty$ and $\dim(X/im(A))<\infty$.
I know by Theorem 4.25 on Rudin's book, I have $$\dim(ker(\lambda I-AB))=\dim(X/im(\lambda I-AB))=\dim(ker(\lambda I-(AB)^*)=\dim(X^*/im(\lambda I-(AB)^*))$$ and $$\dim(ker(\lambda I-BA))=\dim(X/im(\lambda I-BA))=\dim(ker(\lambda I-(BA)^*)=\dim(X^*/im(\lambda I-(BA)^*)).$$
But I am not sure how to get $\dim(ker(A))$ and $\dim(X/im(A))$.
Let $x_n \in \ker A$ and $\|x_n\|\le 1$. Since $(I-BA) x_n = x_n$ and $I-BA$ is compact, a subsequence of $x_n$ converges. Since $\{x_n\} \in \ker A$ is arbitrary, this shows that the closed unit ball in $\ker A$ is compact and and $\ker A$ is finite dimensional.
On the other hand, since $\text{Im}AB \subset \text{Im }A$,
$$ \dim (X/\text{Im}A) \le \dim (X /\text{Im} (AB))$$
Note that $X /\text{Im} (AB)$ is finite dimensional by Theorem 4.25. Thus $\dim (X/\text{Im}A)$ is finite.
FYR, $A, B$ are called Fredholm operators