I have $(x',y')=(x-y-x(x^2+y^2)+\frac{xy}{\sqrt{x^2+y^2}},x+y-y(x^2+y^2)-\frac{x^2}{\sqrt{x^2+y^2}} )$
Now I want to use polar coordinates $(x,y)=(r\cos(t),r\sin(t))$ to get $(r',t')=(r(1-r^2),2\sin(\frac{t}{2})^2)$
I do not see this relation. When I put $x=\cos t$, $y=\sin t$ into the system of differential equations, I only get $(r\cos(t)-r\sin(t)-r^3\cos(t)+r\cos(t)\sin(t),r\cos(t)+r\sin(t)-r^3\cos(t)-r\cos(t)^2)$.
Using $x=r \cos{t}$, $y=r \sin{t}$:
$$x'=(\cos{t}) r' - r (\sin{t}) \, t'$$ $$y'=(\sin{t}) r' + r (\cos{t}) \, t'$$
So we get
$$\left ( \begin{array}\\ \cos{t} & -r \sin{t} \\ \sin{t} & r \cos{t} \end{array} \right ) \left ( \begin{array}\\ r' \\ t' \end{array} \right ) = \left ( \begin{array}\\ r \cos{t} - r \sin{t} - r^3 \cos{t} + \sin{t} \cos{t} \\ r \cos{t} + r \sin{t} - r^3 \sin{t} - \cos^2{t} \end{array} \right ) $$
Multiply both sides by the matrix inverse to get
$$\left ( \begin{array}\\ r' \\ t' \end{array} \right ) = \frac{1}{r} \left ( \begin{array}\\ r\cos{t} & r\sin{t} \\ - \sin{t} & \cos{t} \end{array} \right ) \left ( \begin{array}\\ r \cos{t} - r \sin{t} - r^3 \cos{t} + \sin{t} \cos{t} \\ r \cos{t} + r \sin{t} - r^3 \sin{t} - \cos^2{t} \end{array} \right ) $$
Just do out the multiplication. It is messy, but there is a lot of cancellation and we get
$$\left ( \begin{array}\\ r' \\ t' \end{array} \right ) = \left ( \begin{array}\\ r-r^3 \\ 1 - \cos{t} \end{array} \right ) $$