I have two matrices:
$$ A = \begin{bmatrix} 1 & 2 & -1 \\\ 0 & 1 & 0 \\\ 1 & 0 & 3 \end{bmatrix}, B = \begin{bmatrix} 1 & 0 & 0 \\\ 0 & 2 & 1 \\\ 0 & 0 & 2 \end{bmatrix} $$
To show that the two matrices are similar, I first found the eigenvalues of matrix A
$$ \begin{align*} A & = \begin{bmatrix} 1 & 2 & -1 \\\ 0 & 1 & 0 \\\ 1 & 0 & 3 \end{bmatrix} \\\ |A - λE| & = λ^3 - 5λ^2 + (3+4-1)λ-(3+1) \\\ & = λ^3-5λ^2+8λ-4 \\\ & = (λ-1)(λ-2)^2 = 0 \\\ |B - μ E| & = (μ-1)(μ-2)^2 = 0 \end{align*} $$
Their eigenvalues of $A$ and $B$ are the same, therefore the sufficient condition of similarity of the matrix is established. Also, I have tried multiple times writing the matrix with the form $B = P^{-1}AP$. Yet, interestingly, I still failed to get the form just as matrix $B$.
The procedure that I took on matrix A is:
$$ \begin{align*} & T_A(1) = \begin{bmatrix} 0 & 2 & -1 \\\ 0 & 0 & 0 \\\ 1 & 0 & 2 \end{bmatrix} \begin{bmatrix} x \\\ y \\\ z \end{bmatrix} = \begin{bmatrix} 0 \\\ 0 \\\ 0 \end{bmatrix} \\\ & u_1 = \begin{bmatrix} -4 \\\ 1 \\\ 2 \end{bmatrix} \\\ & T_A(2) = \begin{bmatrix} -1 & 2 & -1 \\\ 0 & -1 & 0 \\\ 1 & 0 & 1 \end{bmatrix} \begin{bmatrix} x \\\ y \\\ z \end{bmatrix} = \begin{bmatrix} 0 \\\ 0 \\\ 0 \end{bmatrix} \\\ & u_2 = \begin{bmatrix} -1 \\\ 0 \\\ 1 \end{bmatrix}, u_3 = \begin{bmatrix} 1 \\\ 2 \\\ 1 \end{bmatrix} \end{align*} $$
The $u_3$ is actually the cross product of $u_1, u_2$. The eigenvalue $2$ has an algebraic multiplicity of 2, yet its eigenvector is only $1$, this tells me that the matrix A is not diagonalizable. What did I mess up here?
Based on the OP and comments suggested, the matrix can be found,
$$ \begin{align*} & A[p_1,p_2,p_3] = [p_1,p_2,p_3]B \\\ & A[p_1,p_2,p_3] = [p_1,2p_2,p_2+2p_3] \\\ \end{align*} $$
$$ \begin{align*} \begin{cases} Ap_1 = p_1 \\\ Ap_2 = 2p_2 \\\ Ap_3 = p_2+2p_3\end{cases} \end{align*} $$
$$ \begin{align*} & (A-E)p_1 = 0 \\\ & (A-E)p_1 = \begin{bmatrix} 0 & 2 & -1 \\\ 0 & 0 & 0 \\\ 1 & 0 & 2 \end{bmatrix} \\\ & p_1 = \begin{bmatrix} -4 \\\ 1 \\\ 2 \end{bmatrix} \end{align*} $$
$$ \begin{align*} & (A-2E)p_2 = 0 \\\ & (A-2E)p_2 = \begin{bmatrix} -1 & 2 & -1 \\\ 0 & -1 & 0 \\\ 1 & 0 & 1 \end{bmatrix} \\\ & p_2 = \begin{bmatrix} 1 \\\ 0 \\\ -1 \end{bmatrix} \end{align*} $$
$$ \begin{align*} & (A-2E)p_3 = p_2 \\\ & (A-2E)p_3 = \begin{bmatrix} -1 & 2 & -1 \\\ 0 & -1 & 0 \\\ 1 & 0 & 1 \end{bmatrix} = \begin{bmatrix} 1 \\\ 0 \\\ -1 \end{bmatrix} \\\ & p_3 = \begin{bmatrix} -1/2 \\\ 0 \\\ -1/2 \end{bmatrix} \end{align*} $$
$$ \begin{align*} B = P^{-1}AP = \begin{bmatrix} 1 & 0 & 0 \\\ 0 & 2 & 1 \\\ 0 & 0 & 2 \end{bmatrix} \end{align*} $$
What I noticed is that the vectors should not be scaled in this case, especially the third column.