Let $$A = \begin{pmatrix} 1 & -3 & 0 \\ 3 & 4 & -3 \\ 3 & 3 & -2\end{pmatrix}$$ Find an invertible $P \in M_3(\mathbb{R})$ such that $$B = P^{-1}AP = \begin{pmatrix} 1 & 0 & 0 \\ 0 & 1 & 3 \\ 0 & -3 & 1\end{pmatrix}$$
The problem is that the characteristic polynomial of both $A$ and $B$ is $-(\lambda-1)(\lambda^2 - 2\lambda+10)$, which has imaginary root, so I can't just straight up find the diagonalization decomposition of $A$ and $B$ and multiply them, as $P$ might contain complex number, which isn't what I need.
Rational canonical form might work, but that is beyond my ability.
Any solution/helps would be appreciated.
Let $P=\begin{pmatrix} 1 & 0 & 1 \\ 0 & 1 & 0 \\ 1 & 1 & 0 \end{pmatrix}$. $\;$ You can check that $P^{-1}AP=\begin{pmatrix} 1 & 0 & 0 \\ 0 & 1 & 3 \\ 0 & -3 & 1 \end{pmatrix}=B$.
The way I found this matrix was as follows: in order to have $P^{-1}AP=B$, $P$ must send the standard basis vector $e_1$ to an eigenvector of the eigenvalue $1$. The eigenvectors for the eigenvalue $1$ are elements of $\ker(A-I)$. Note that $A-I$ is what you get when you plug $A$ into $x-1$.
Note that $A-I=\begin{pmatrix} 0 & -3 & 0 \\ 3 & 3 & -3 \\ 3 & 3 & -3 \end{pmatrix}$, $\;$ and $\text{rref}(A-I)=\begin{pmatrix} 1 & 0 & -1 \\ 0 & 1 & 0 \\ 0 & 0 & 0 \end{pmatrix}$.
So I let $P(e_1)=\begin{pmatrix} 1 \\ 0 \\ 1 \end{pmatrix}$.
It remains to find out where $P$ should send $e_2$ and $e_3$. I found these vectors as follows: since the characteristic polynomial of $A$ is
$$c_A(x)=(x-1)(x^2-2x+10)$$
I found $P(e_2)$, $P(e_3)$ by finding a basis for $\ker(A^2-2A+10I)$. Note that $A^2-2A+10I$ is what you get when you plug $A$ into $x^2-2x+10$.
Note that $A^2-2A+10I=\begin{pmatrix} 0 & -9 & 9 \\ 0 & 0 & 0 \\ 0 & -9 & 9 \end{pmatrix}$, $\;$ and $\text{rref}(A^2-2A+10I)=\begin{pmatrix} 0 & 1 & -1 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \end{pmatrix}$.
So I let $P(e_2)=\begin{pmatrix} 0 \\ 1 \\ 1 \end{pmatrix}$ and $P(e_3)=\begin{pmatrix} 1 \\ 0 \\ 0 \end{pmatrix}$. $\;$ Hence $P=\begin{pmatrix} 1 & 0 & 1 \\ 0 & 1 & 0 \\ 1 & 1 & 0 \end{pmatrix}$, $\;$ and $P^{-1}AP=B$.