Consider the system $$ \dot{x}=y,\quad\dot{y}=z,\quad\dot{z}=-x-y-z+x^2+az^3. $$ Check that the zero equilibrium has a pair of pure imaginary eigenvalues. Make a linear coordinate transformation which makes the linear part of the system block diagonal.
First, the linearization matrix at $(0,0,0)$ is $$ M=\begin{pmatrix}0 & 1 & 0\\0 & 0 & 1\\-1 & -1 & -1\end{pmatrix} $$ and the eigenvalues are $\lambda_1=-1,\lambda_2=-i,\lambda_3=i$.
Edit due to Did's help
The transformation matrix is $$ S=\begin{pmatrix}1 & -1 & 0\\-1 & 0 & -1\\1 & 1 & 0\end{pmatrix}. $$ With this, we get $$ S^{-1}MS=\begin{pmatrix}-1 & 0 & 0\\0 & 0 & 1\\0 & -1 & 0\end{pmatrix}=:J $$
So the aim is to transform the system $$ \begin{pmatrix}\dot{x}\\\dot{y}\\\dot{z}\end{pmatrix}=M\cdot\begin{pmatrix}x\\y\\z\end{pmatrix}+\begin{pmatrix}0\\0\\x^2+az^3\end{pmatrix}~~~(1) $$ into the system $$ \begin{pmatrix}\dot{u}\\\dot{v}\\\dot{w}\end{pmatrix}=J\cdot\begin{pmatrix}u\\v\\w\end{pmatrix}+\begin{pmatrix}f(u,v,w)\\g(u,v,w)\\h(u,v,w)\end{pmatrix}~~~(2). $$ How do I get from system (1) to system (2)?
Do I have to compute $$ S^{-1}MS \begin{pmatrix}x\\y\\z\end{pmatrix}+S^{-1}\begin{pmatrix}0\\0\\x^2+az^3\end{pmatrix}S? $$ This would give me the system $$ \begin{pmatrix}\dot{u}\\\dot{v}\\\dot{w}\end{pmatrix}=J\begin{pmatrix}u\\v\\w\end{pmatrix}+\begin{pmatrix}\frac{1}{2}(u^2+aw^3)\\\frac{1}{2}(u^2+aw^3)\\-\frac{1}{2}(u^2+aw^3)\end{pmatrix}. $$
Your determination of $S$ is now correct.
In order to transform system (1) into system (2), multiply with $S^{-1}$ from the left and use that $SS^{-1}$ gives you the identity matrix: $$ S^{-1}\begin{pmatrix}\dot{x}\\\dot{y}\\\dot{z}\end{pmatrix}=S^{-1}MS\cdot S^{-1}\begin{pmatrix}x\\y\\z\end{pmatrix}+S^{-1}\begin{pmatrix}0\\0\\x^2+az^3\end{pmatrix} $$ Now, set $$ \begin{pmatrix}u\\v\\w\end{pmatrix}:=S^{-1}\begin{pmatrix}x\\y\\z\end{pmatrix} $$ and what you get is $$ \begin{pmatrix}\dot{u}\\\dot{v}\\\dot{w}\end{pmatrix}=J\begin{pmatrix}u\\v\\w\end{pmatrix}+S^{-1}\begin{pmatrix}0\\0\\x^2+az^3\end{pmatrix}. $$ Finally to get rid of the $x$- and $z$ terms in the last summand, use that $$ \begin{pmatrix}x\\y\\z\end{pmatrix}=S\begin{pmatrix}u\\v\\w\end{pmatrix}. $$ The result is $$ \begin{pmatrix}\dot{u}\\\dot{v}\\\dot{w}\end{pmatrix}=J\begin{pmatrix}u\\v\\w\end{pmatrix}+\begin{pmatrix}\frac{1}{2}\cdot ((u-v)^2+a(u+v)^3)\\\frac{1}{2}\cdot ((u-v)^2+a(u+v)^3)\\-\frac{1}{2}\cdot ((u-v)^2+a(u+v)^3)\end{pmatrix} $$ so you've got your transformed ODE-System.