I have this ODE system:
$\frac{\mathrm{d} x}{\mathrm{d} t}=x-y-(x^{2}+y^{2})x$
$\frac{\mathrm{d} y}{\mathrm{d} t}=x+y-(x^{2}+y^{2})y$
My thought is, firstly make variable change $a=x^{2}$ and $b=y^{2}$, then the system becomes
$\frac{1}{2}a^{2}=a-xy-(a+b)a$
$\frac{1}{2}b^{2}=b+xy-(a+b)b$
Add them together, we get
$\frac{1}{2}(\dot{a}+\dot{b})=a+b-(a+b)^2$
Let $w=a+b$, it becomes $\frac{1}{2}\dot{w}=w-(w)^2$, which can be solved. But we can only know the behavior of $w=a+b=x^{2}+y^{2}$ by this method, I would like to solve it for both $x$ and $y$. I'm not sure I'm on the right truck, can anyone help me?
One option is to plug in what you described for the factor $x^2+y^2$ into the system and then solve it.
Another option is to go to polar coordinates up front, so you get
$$\dot{r} \cos(\theta) - r \sin(\theta) \dot{\theta} = r(\cos(\theta)-\sin(\theta)) + r^3 \cos(\theta) \\ \dot{r} \sin(\theta) + r \cos(\theta) \dot{\theta} = r(\cos(\theta)+\sin(\theta)) + r^3 \sin(\theta).$$
From here you can invert $\begin{bmatrix} \cos(\theta) & -r\sin(\theta) \\ \sin(\theta) & r\cos(\theta) \end{bmatrix}$, the inverse is $ \begin{bmatrix} \cos(\theta) & \sin(\theta) \\ -\sin(\theta)/r & \cos(\theta)/r \end{bmatrix}$. Multiplying both sides by that matrix yields a simple system for $\dot{r}$ and $\dot{\theta}$ (they end up being decoupled, actually).