Bring the system $$\begin{cases} x'=y+xz,\\ y'=x^2+y^2+z^2,\\z'=-2z+xy \end{cases}$$ to a normal form up to the second order (kill all non-resonant quadratic terms).
The equilibrium is $(0,0,0)$ and the linearization matrix is $$ A=\begin{pmatrix}0 & 1 & 0\\0 & 0 & 0\\0 & 0 & -2\end{pmatrix}, $$ hence the eigenvalues are $$ \lambda_1=\lambda_2=0,\lambda_3=-2. $$
- In the first equation, the term $xz$ is non-resonant.
- In the second equation, the term $z^2$ is non-resonant.
- In the third equation, the term $xy$ is non-resonant.
Edit due to Jonas' help in the comments
Let $$\begin{cases} X=x+a_1y^2+a_2yz+a_3z^2,\\ Y=y+b_1x^2+b_2xz+b_3 z^2,\\ Z=z+c_1x^2+c_2xy+c_3 y^2\end{cases} $$
Put this into the ODEs, i.e. $$\begin{cases} X'=Y+XZ=y+b_1x^2+b_2xz+b_3z^2+xz+\text{ terms of 3rd order or higher}&(1)\\ Y'=X^2+Y^2+Z^2=x^2+y^2+z^2+\text{ terms of 3rd order or higher}&(2)\\ Z'=-2z-2c_1x^2-2c_2xy-2c_3y^2+xy+\text{ terms of 3rd order or higher}&(3). \end{cases}$$
From (1), we get that $$ b_1=b_3=0, b_2=-1. $$
But I do not know how to get the other coeffcients $a_i, c_i, i=1,2,3$ from (2) and (3).
What is below is done by brute force. It is not what the theory recommends (neither it is useful in general), but it is what you ask.
Write $$x=X+a_1X^2+b_1Y^2+c_1Z^2+d_1XY+e_1XZ+f_1YZ,$$ $$y=Y+a_2X^2+b_2Y^2+c_2Z^2+d_2XY+e_2XZ+f_2YZ,$$ $$z=Z+a_3X^2+b_3Y^2+c_3Z^2+d_3XY+e_3XZ+f_3YZ.$$ Then $$x'=X'+2a_1XX'+2b_1YY'+2c_1ZZ'+\cdots$$ while $$ \begin{split} y+xz &=Y+a_2X^2+b_2Y^2+c_2Z^2+d_2XY+e_2XZ+f_2YZ+\\ &\quad(X+a_1X^2+b_1Y^2+c_1Z^2+d_1XY+e_1XZ+f_1YZ)\times\\ &\quad(Z+a_3X^2+b_3Y^2+c_3Z^2+d_3XY+e_3XZ+f_3YZ). \end{split} $$ You do the same for the other two equations. Since $x'=y+xz$, etc, you obtain $3$ equations that you can use to find $X',Y',Z'$. You need to express the right-hand sides as Taylor series, after which you will get for example$$X'=Y+(\text{expression in terms of the constants})YZ+\cdots.$$ It is the expression in the line above (and all others that you know that may be erased) that give equations to find the (possible) constants.
In practice this is not very effective and it is much better to solve the homological equations. For that, again I recommend the book by Guckenheimer and Holmes.