Let $A, B$ be $4\times 4$ square matrix given by,
$$A=\begin{bmatrix}0 &0&1&0\\0&0&0&1\\0&0&0&0\\0&0&0&0\\\end{bmatrix}, B=\begin{bmatrix}0 &0&1&0\\0&0&0&0\\0&0&0&1\\0&0&0&0\\\end{bmatrix}$$
\begin{array}{|c|c|c|} \hline \text{Property}& A & B\\\hline \text{Determinant}& 0& 0\\ \hline \text{Trace}& 0 &0 \\ \hline \text{Rank}& 2 & 2\\ \hline \text{Char. Polynomial}& x^4 &x^4\\ \hline \end{array}
I want to show that $A$ and $B$ are not similar. All the problems I encountered so far were usually solved by showing that one of these above mentioned properties does not match.
How to approach this?
According to the comments, $A^2,B^2$ are not similar because
$\begin{pmatrix}1&0\\0&0\end{pmatrix}\begin{pmatrix}0&1\\0&0\end{pmatrix}\not= 0$.Yet, even if the previous product had been zero, $A,B$ would not have been similar. More generally
let $A=\begin{pmatrix}0&M\\0&0\end{pmatrix},B=\begin{pmatrix}0&U\\0&V\end{pmatrix}$ where each block has dimension $n\times n$, $M$ is invertible and $V\not=0$. Then $A,B$ are not similar (even if $V^2=0,UV=0$).
Proof. We show that if $Z=\begin{pmatrix}P&Q\\R&S\end{pmatrix}$ and $AZ=ZB$, then $\det(Z)=0$.
We obtain $MR=0$ and then $R=0$;
$SV=0$, then $\det(S)=0$;
$\det(Z)=\det(P)\det(S)=0$. $\square$