Let $A$ and $B$ be matrices over the real numbers, such that $A$ is $3 \times 5$ and $B$ is $5 \times 3$, and the product $AB$ is $$ \left( \begin{array} \\ 1& 1 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{array} \right). $$ I want to find the Jordan canonical form of $BA$.
I am rather stuck on this. The only information that I know to go on is that $BA$ is $5 \times 5$ and it's rank is at most 3, so $0$ is an eigenvalue of it and it's multiplicity is at least $2$.
On
As you noted, we have $$r(BA) \le \max\{r(A),r(B)\} \le 3$$ so $0$ is an eigenvalue of $BA$ with algebraic multiplicity at least $2$.
Clearly $\sigma(AB) = \{1\}$ and $\dim\ker (AB-I) = 2$. If $(AB-I)x = 0$ then $$(BA-I)Bx = B(AB-I)x = 0$$ which implies that $B(\ker(AB-I)) \subseteq \ker (BA-I)$. From $$\ker B\cap \ker (AB-I) = \{0\}$$ we get that $B|_{\ker(BA-I)}$ is injective. Therefore $$\dim\ker (BA-I) \ge \dim B(\ker(AB-I)) = \dim\ker(AB-I)=2.$$
Similarly we get $A(\ker(BA-I))\subseteq \ker(AB-I)$ and $A|_{\ker(AB-I)}$ is injective so $$2 = \dim \ker(AB-I) \ge \dim A(\ker(BA-I)) = \dim \ker(BA-I)$$ from which we conclude $\dim \ker(BA-I)=2$.
Since $0$ and $1$ are eigenvalues of $BA$ with geometric multiplicities $\ge 2$, the spectrum of $BA$ with algebraic multiplicities is $$\sigma(BA) = \{0,0,1,1,?\}.$$ To determine the last eigenvalue note that $$2\,+\,?=\operatorname{Tr}(BA) = \operatorname{Tr}(AB)=3$$ so the last eigenvalue is also $1$. Hence it must be $\dim\ker(BA) = 2$ so there are precisely two $0$-blocks of size $1$. From $\dim\ker(BA-I) = 2$ we get that there are two $1$-blocks and that $BA$ is not diagonalizable so there is one $1$-block of size one and one $1$-block of size $2$: $$\begin{bmatrix} 1 & 1 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 & 0 \\ 0 & 0 & 1 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0\end{bmatrix}.$$
On
Let $O_1, O_2, O_3$ are $2\times5, 5\times2, 2\times 2$ zero matrices, respectively. Define $$ A_1=\binom{A}{0_1}, B_1=(B, 0_2) $$ be $5\times 5$ matrices. Then $$ B_1A_1=BA, A_1B_1=\bigg(\begin{matrix}AB&0_2\\0_1&0_3\end{matrix}\bigg).$$ By $$ \det(B_1A_1-\lambda I_5)=\det(A_1B_1-\lambda I_5) $$ one has $$ \det(BA-\lambda I_5)=\det\bigg[\bigg(\begin{matrix}AB&0_2\\0_1&0_3\end{matrix}\bigg)-\lambda I_5\bigg]=\lambda^2\det(AB-\lambda I_3)=\lambda^2(\lambda-1)^3. $$ This implies that $BA$ has eigenvalues $0$ and $1$. Clearly $BA$ has eigenvectors $v_4=(0,0,0,1,0)^T,v_5=(0,0,0,0,1)^T$ of eigenvalue $0$ since $$ (BA)v_4=B_1A_1v_4=0, (BA)v_5=B_1A_1v_5=0. $$ Let $w_1=(1,0,0)^T,w_2=(0,1,0)^T,w_1=(0,0,0)^T$ and then $$ ABw_1=v_1,ABw_2=w_1,ABw_3=w_3, $$ namely $AB$ has eigenvectors $w_1,w_3$ are of eigenvalue 1 and $w_2$ as a generalized eigenvector of 1. Define $$ v_1=Bw_1,v_2=Bw_2,v_3=Bw_3. $$ Then $$ (BA)v_1=B(ABw_1)=Bw_1=v_1, (BA)v_2=B(ABw_2)=Bw_2=v_1, (BA)v_3=B(ABw_3)=Bw_3=v_3 $$ namely $BA$ has eigenvectors $v_1,v_3$ are of eigenvalue 1 and $v_2$ as a generalized eigenvector of 1. Let $$ S=(v_1,v_2,v_3,v_4,v_5) $$ and then $$ S^{-1}(BA)S=\left(\begin{matrix} 1&1&0&0&0\\ 0&1&0&0&0\\ 0&0&1&0&0\\ 0&0&0&0&0\\ 0&0&0&0&0\end{matrix}\right). $$
If $ABv=\lambda v$, then $BA(Bv)=B(ABv)=B(\lambda v)=\lambda(Bv)$, so if $v$ is an eigenvector of $AB$ with eigenvalue $\lambda$, then $Bv$ is an eigenvector of $BA$ with eigenvalue $\lambda$. If $\lambda\neq 0$, then $Bv\neq 0$ because $A(Bv)\neq 0$, so this shows that, with the exception of $0$, $AB$ and $BA$ have the same eigenvalues with the same geometric multiplicities.
Similarly, one can show that the characteristic polynomials of $AB$ and $BA$ are the same up to an additional factor of $t^k$, so the algebraic multiplicities of the eigenvalues are the same, except for $0$.
In low dimensions, this information is enough to determine the Jordan Normal Form, except for the $0$ block.
For that, we can look at the minimal polynomial. We have that $(AB-I)^2=0$, and so $B(AB-I)^2A=(BA-I)^2BA=0$. Because this has a single factor of $BA$, we cannot have a block of $0$ of size bigger than $1$.
This argument actually shows that $AB$ and $BA$ have the same sized largest block for each eigenvalue except for $0$, and the maximum block size for $0$ can differ by at most $1$.
To see that the maximal block size for $0$ can change, consider $A=(0,1)$, $B=(1,0)^T$.
As an alternative (and somewhat more useful way, as it will give results in higher dimensions) to looking at characteristic polynomials. We have by induction that $B(AB-\lambda I)^k=(BA-\lambda I)^k B$, and so if $v$ is an approximate eigenvector of $AB$ with eigenvalue $\lambda$ and order $k$, then $Bv$ is an approximate eigenvector of $BA$ with eigenvalue $\lambda$ and order $k$. I think this can be used to show that when $\lambda\neq 0$, the approximate eigenspaces have the same dimensions, and so all the block sizes will be the same for the non-zero eigenvalues. However, I have not checked the details, so this may not go through.