$$A= \left( \begin{array}{ccc|c} 5&3&\sqrt2\\ 0&1&0\\ 0&0&5\\ \end{array} \right) , \qquad B= \left( \begin{array}{ccc|c} 5&0&0\\ 0&1&0\\ 0&0&5\\ \end{array} \right) , \qquad C= \left( \begin{array}{ccc|c} 1&0&0\\ 3&5&0\\ \sqrt2&0&5\\ \end{array} \right) $$ The correct answer is that $B$ and $C$ are similar.
First, I checked the trace of each matrix
$$ \mbox{tr} A=5+1+5=11, \qquad \mbox{tr} B=5+1+5=11, \qquad \mbox{tr} C=1+5+5=11 $$
Then, I checked ranks and got $\mbox{rank}(A)=\mbox{rank}(B)=\mbox{rank}(C)=3$. Then, also for determinants
$$\det(A)=\det(B)=\det(C)=25$$
Then, the characteristic polynomial
$$\det (\lambda I -A)=\det (\lambda I -B)=\det (\lambda I -C)= (\lambda-5)^2 \cdot (\lambda-1)$$
$\lambda = 5$: algebraic multiplicity = 2, geometric multiplicity = 2
$\lambda = 1$: algebraic multiplicity = 1, geometric multiplicity = 1 (because the algebraic multiplicity of 1 is 1)
My conclusion is that all of them are similar and not just $B$ and $C$. Am I missing something?
That is not correct. Yes, matrices $B$ and $C$ are similar. But the eigenspace corresponding to the eigenvalues $5$ is only $1$-dimensional in the case of matrix $A$ (it is spanned by $(1,0,0)$); in other words, the geometric multiplicity of the eigenvalue $5$ is $1$ in the case of matrix $A$. But it is $2$ for $B$ and $C$.