Let the endomorphism $f$ be given by $f: \mathbb{R^3} \to \mathbb{R^3}, a \to \begin{pmatrix} 1 & 1 & 2\\ 0 & 1 & 0 \\ 3 & 0 & 1 \end{pmatrix} \cdot a$.
Find a basis for Fix($f$).
So we need to solve
$ \begin{pmatrix} 1 & 1 & 2\\ 0 & 1 & 0 \\ 3 & 0 & 1 \end{pmatrix} \cdot \begin{pmatrix} x \\ y \\ z \end{pmatrix} = \begin{pmatrix} x \\ y \\ z \end{pmatrix}$
This gives
$\begin{pmatrix} x + y + 2z \\ y \\ 3x + z \end{pmatrix} = \begin{pmatrix} x \\ y \\ z \end{pmatrix}$
Using the third equation $3x + z = z$, we get $3x = 0 \implies x = 0$.
Using the first equation $x + y + 2z = x$, we get $y + 2z = 0 \implies y = -2z$. So we have $\begin{pmatrix} 0 \\ -2z \\ z \end{pmatrix} = z \begin{pmatrix} 0 \\ -2 \\ 1 \end{pmatrix}$
So a basis for Fix($f$) would be $\begin{pmatrix} 0 \\ -2 \\ 1 \end{pmatrix}$. Is that correct ?