By direct calculation show that (using polar coordianted) that $$ \dot x=x-y-x(x²+y²)\\ \dot y=x+y-y(x²+y²) $$ has a limit cycle.
I need help understanding how to test whether there is a limit cycle or not. Do we check whether $$ f_x+g_y=0? $$ I am not sure whether this should be done once it is converted to polar series or before. Nor can I find a proper source about "how to test whether a system has a limit cycle" on the web.
On
Consider new variable $z=x+iy$. In this variable your systems reads $$ \dot z=(1+i)z-z|z|^2. $$ Passing to the polar coordinates $z(t)=\rho(t)e^{i\varphi(t)}$ you will find \begin{align} \dot \rho&=\rho(1-\rho^2)\\ \dot \varphi&=1. \end{align} Note that the equations are decoupled and conclude that 1) the first equation has a solution $\rho=1$ -- which means this is a closed trajectory on the phase plane 2) there are no other closed solutions in a small neighborhood of this orbit -- which means that your closed trajectory is a limit cycle.
Unfortunately there are no general methods to detect limit cycles. What you wrote concerns the conditions of non-existence of limit cycles. In textbook problems it is often useful to try polar coordinates. Another important thing to know is the Poincare--Bendixson theory. An extremely good book addressing the issues of finding periodic solutions to systems of ODE is Periodic Motions by Miklos Farkas.
Hint: Consider $z=x^2+y^2$. Show that $\dot z=U(z)$ for some explicit function $U$. Find the zeroes of $U$ and deduce that either $z(t)=0$ for every time $t$, or that $z(t)\to1$ when $t\to\infty$. The circle of equation $z=1$ is is your limit cycle.