Let $V$ be a nonzero, finite-dimensional vector space over a field $F$, and assume that the characteristic of $F$ is not $2$. Let $A, B$ be endomorphisms of $V$ such that $A^2=B^3=(AB)^2=I$ (the identity), and $Bx\ne x$ for all nonzero vectors $x\in V$. Show that there is a $2$-dimensional subspace $W$ of $V$ such that $A(W)\subset W$ and $B(W)\subset W$.
I am able to show that $A(W)\subset W$ for some $2$-dimensional subspace $W$. Indeed, choose $\alpha\in V$ and $\alpha\ne 0$, if $\alpha, A\alpha$ are linear independent, then we choose $W=\text{Span}\{\alpha,A\alpha\}$ and we are through, since $A^2=I$. Otherwise, choose $\beta\notin\text{Span}\{\alpha\}$ and either $W=\text{Span}\{\beta, A\beta\}$ or $W=\text{Span}\{\alpha,\beta\}$ satisfies $A(W)\subset W$.
However, I have no idea how to show the same thing for $B$. I know that we can also find such a $2$-dimensional subspace for $AB$, but the conditions: $B^3=I$ and $Bx\ne x$ for all nonzero $x\in V$ really confuse me. Can someone give me a hint? Thank you.
What follows is a proof when $K$ is algebraically closed.
The hypothesis $1\notin \mathrm{Spec}(B)$ seems to be useless, but it simplifies the proof. Then one has $B^2+B+I_n=0_n$ (that is convenient if the characteristic of $K$ is not $3$) and we assume that $n\geq 3$.
Let $\mathcal{A}$ be the unitary algebra generated by $A,B$. Since $ABA=-B-I$ and $BAB=A$, $\mathcal{A}=\mathrm{span}(I,A,B,AB,BA)$ and is not whole $M_n(K)$. According to the Burnside's theorem, $A,B$ have a common proper invariant subspace of dimension $r$. Up to a change of basis, we may assume
$$A=\begin{pmatrix}P_r&Q\\0&R_{n-r}\end{pmatrix},\ B=\begin{pmatrix}U_r&V\\0&W_{n-r}\end{pmatrix}$$ where $1\leq r<n$. Notice that $(AB)^2=I$ implies that $(BA)^2=I$ and $(AB-BA)^2=2I-AB^2A-BA^2B=3I$. Then $AB-BA$ is invertible and $r>1$. If $r=2$, then we are done; otherwise, $r\geq 3$ and $P^2=(PU)^2=I_r,U^2+U+I_r=0$ (the same hypothesis as above with $r<n$). We conclude by a recurrence reasoning.