Let $A$ and $B$ be $3\times3$ square matrices (maybe could also examine $n\times n$) and let $U$ be a subspace where $A,B,AB\in U$. All matrices in $U$ are nilpotent. I conjecture that all matrices in the subspace either commute up to a constant ($AB=\alpha BA$), or they are all strictly upper (or all strictly lower) matrices.
My reasoning is this: if $A\in U$ and $B\in U$ are nilpotent and commute up to a constant, or are strictly upper (strictly lower) matrices, then $AB$ and $A+B$ will also be nilpotent. Now of course you could have matrices $A$ and $B$ that do not commute etc. but are such that $AB$ and $A+B$ are nilpotent. But the formulation of the subspace would need $ABA$, $AB^2$, $BAB$, $(A+B)A$, $AB+BA$ and infinitely more to be nilpotent. I just can't see how any $3 \times 3$ matrix (even $n \times n$) could be in the subspace without fulfilling my conjecture. Maybe there is an obvious proof of my conjecture, or maybe I am just wrong and their exists an obvious counter example, or maybe it is a hard problem. Either way I would love to get a response, thanks!
Edit: My knowledge in pure math (such as abstract algebra) is severly limited, nevertheless if there exists a proof using tools from such areas I would still appreciate it a lot!
If $U$ an algebra of $3\times3$ nilpotent matrices over some field $F$, there exists an invertible matrix $P$ such that $P^{-1}UP$ is a subspace of strictly upper triangular matrices.
Here is the proof. Suppose first that $U$ has a member $A$ of rank $2$. By a change of basis, we may assume that $A$ is the nilpotent Jordan block. Now let $B\in U$. By assumption, $$ A^2B=\pmatrix{b_{31}&b_{32}&b_{33}\\ 0&0&0\\ 0&0&0}, \ AB=\pmatrix{b_{21}&b_{22}&b_{23}\\ b_{31}&b_{32}&b_{33}\\ 0&0&0} \ \text{and}\ B=\pmatrix{b_{11}&b_{12}&b_{13}\\ b_{21}&b_{22}&b_{23}\\ b_{31}&b_{32}&b_{33}} $$ are nilpotent. That $A^2B$ is nilpotent implies that $b_{31}=0$. In turn, that $AB$ is nilpotent implies that $b_{21}=b_{32}=0$. Consequently, since $B$ is nilpotent, we must also have $b_{11}=b_{22}=b_{33}=0$. Hence $B$ is strictly upper triangular.
Now suppose that all nonzero members of $U$ are rank-one nilpotent matrices. Let $A$ be a nonzero member of $U$. By a change of basis, we may assume that $A=e_1e_3^T$. Since $U$ is a subspace of rank-one matrices, either $U\subseteq\{e_1x^T:x\in F^3\}$ or $U\subseteq\{xe_3^T:x\in F^3\}$. In the former case, for an arbitrary $B=e_1x^T\in U$ to be nilpotent, $x_1$ must be zero. Hence $B$ is strictly upper triangular. Similarly, in the latter case, for $B=xe_3^T$ to be nilpotent, $x_3$ must be zero. Hence $B$ is strictly upper triangular.