Determine up to similarity all $3 \times 3$ complex matrices $A$ such that $A^4 + 2A^3 + A^2 = 0$ and $A^2 + A \neq 0$. Give the characteristic and minimal polynomial of each matrix.
I'm not quite sure how to approach this problem. First, it's clear that the minimal polynomial of any matrix which satisfies these conditions must divide, \begin{align*} p(x) = x^4 + 2x^3 + x^2 = 0 \end{align*}
and one can factor this into $x^2(x+1)(x+1)$. Also, I am aware that similar matrices share the same characteristic/minimal polynomials. There may also, perhaps, be a connection to Jordan Canonical Form. What I am not sure about is, how can we find the characteristic polynomial, how exactly does the condition $A^2 + A \neq 0$ factor into this, and how do I know when I've found all of the matrices.
To make the answer complete I first repeat some of my comment. Suppose $J$ is the Jordan's canonical form of $A$, with $J=PAP^{-1}$. Then we have \begin{align*} &A^4 + 2A^3 + A^2 = 0 &\\ \iff & P(A^4 + 2A^3 + A^2 )P^{-1}= 0 & (\text{since $P0P^{-1}=0$})\\ \iff & J^4 + 2J^3 + J^2 = 0 & (\text{since $PA^nP^{-1}=PAP^{-1}PAP^{-1}\cdots PAP^{-1}=J^n$}) \end{align*} The same argument gives $J^2+J\ne 0$. So it suffices to determine all the Jordan-canonical form $J$ s.t. $J^4 + 2J^3 + J^2 = 0$, $J^2+J\ne 0$. There are three cases according to the sizes and numbers Jordan blocks. We are going to check them case by case.
For convinience denote $f(x)=x^4+2x^3+x^2$, $ g(x)=x^2+x $. Note that $f(x_0)=0$ $\implies$ $x_0\in\{0,-1\}$ $\implies$ $g(x_0)=0$.
Case 1: $J$ has Jordan blocks "$1+1+1$". (That is it has three Jordan blocks, each with size one.)
Then $J=\mathtt{diag}\{\lambda,\mu, \omega\}$, where $\lambda,\mu, \omega$ need not be distinct. Since $J^4 + 2J^3 + J^2 = 0$, we have $f(\lambda)=f(\mu)=f(\omega)=0$. Therefore $\lambda,\mu, \omega\in\{0,-1\}$. But this implies $\lambda,\mu, \omega$ are zeros of $g(x)$. Hence $J^2+J=\mathtt{diag}\{g(\lambda),g(\mu), g(\omega)\}=0$, a contradiction. Therefore Case 1 is impossible.
Case 2: $J$ has Jordan blocks "$2+1$".
Then \begin{equation*} J=\begin{pmatrix} \lambda & 1 & \\ & \lambda & \\ & & \mu \end{pmatrix}, \end{equation*} where $\lambda,\mu$ need not be distinct. A direct computation gives \begin{equation*} J^4 + 2J^3 + J^2 =\begin{pmatrix} f(\lambda) & 4\lambda^3+6\lambda^3+2\lambda^2 & \\ & f(\lambda) & \\ & & f(\mu) \end{pmatrix}. \end{equation*} So we must have $f(\lambda)=f(\mu)=4\lambda^3+6\lambda^3+2\lambda^2=0$. This gives $\lambda,\mu \in\{-1,0\}$.
Another direct computation gives \begin{equation*} J^2+J=\begin{pmatrix} \lambda^2+\lambda & 2\lambda+1 & \\ & \lambda^2+\lambda & \\ & & \mu^2+\mu \end{pmatrix}. \end{equation*} When $\lambda\in\{0,-1\}$, $2\lambda+1\ne 0$. Therefore $J^2+J\ne 0$ whenever $\lambda\in\{0,-1\}$.
Hence case 2 gives four solutions: \begin{equation*} J=\begin{pmatrix} \lambda & 1 & \\ & \lambda & \\ & & \mu \end{pmatrix}, \end{equation*} where $\lambda,\mu\in\{0,-1\}$.
Case three: $J$ has Jordan blocks "$3$". Then \begin{equation*} J=\begin{pmatrix} \lambda & 1 & \\ & \lambda & 1 \\ & & \lambda \end{pmatrix} . \end{equation*} A direct proof gives \begin{equation*} J^4 + 2J^3 + J^2=\begin{pmatrix} f(\lambda) & \text{the value here is not important} & 4\lambda^2+3\lambda+1 \\ & f(\lambda) & \text{the value here is not important} \\ & & f(\lambda) \end{pmatrix} . \end{equation*} Since $f(\lambda)$ and $ 4\lambda^2+3\lambda+1 $ can not be zero at the same time, we conclude Case three is impossible.
To sum up there are (up to similarity) four choices for $J$ (and hence for $A$): \begin{equation*} J=\begin{pmatrix} \lambda & 1 & \\ & \lambda & \\ & & \mu \end{pmatrix}, \end{equation*} where $\lambda,\mu\in\{0,-1\}$.