Show that there exists a limit cycle by finding a positive invariant set $$y'' = -y + y'(1-3y^2-2y')^2$$
I rewrote this second order ODE as a system of two first order ODE's
$y_1' = y_2$ and $y_2' = -y_1 + y_2(1-3y_1^2-2y_2)^2$
Then I rewrote the system in polar co-ordinates:
$r' = r\sin^2\theta(1-2r^2\cos^2\theta-2r\sin\theta)^2$ and $\theta' = 1 + \sin\theta(1-3r^2\cos^2\theta-2r\sin\theta)\cos\theta$
Then I used the fact that $0 \leq \sin^2\theta \leq 1$ to get $0 \leq r' \leq r(1-2r^2\cos^2\theta-2r\sin\theta)$ and then the fact that $0 \leq \cos^2\theta \leq 1$ to get $$r(1-2r\sin\theta)^2 \leq r' \leq r(1-3r^2 -2r\sin\theta)^2$$ and then I used the fact that $-1 \leq \sin\theta \leq 1$ to get $$r(1-3r^2 + 2r)^2 \leq r' \leq r(1-2r)$$
From here I'm not sure how to proceed to find this positive invariant set. The solutions to this question state that the positive invariant set is given by $$D = \left\{ r \ : \frac{1}{3} \leq r^2 \leq \frac{1}{2}\right\}$$
And I'm not sure how to arrive at that.
The task as reproduced has to be wrong. There has to be a typo either in the original task or in its transmission to the question. In the given version, one gets, as observed, that $\dot V\ge 0$ everywhere, using $V=y^2+y'^2$. Furthermore, there is no radius with $\dot V<0$ on the whole circle of that radius, as required for the existence of a trapping, or positive invariant annulus, so that there are no closed orbits and thus no limit cycles.
With a slight modification, exchanging the closing parentheses and the square to $$ y''+y=y'(1-3y^2-2y'^2) $$ one gets \begin{align} \dot V=\frac{d}{dt}V(y(t),y'(t))=\frac d{dt}(y'^2+y^2) &=2y'(y''+y) \\ &=2y'^2(1-3y^2-2y'^2)\\[1em] &=2y'^2(1-3(y^2+y'^2)+y'^2)\tag{1}\\[1em] &=2y'^2(1-2(y^2+y'^2)-y^2).\tag{2} \end{align} This is (1) non-negative for $V=y^2+y'^2\le\frac13$ and (2) non-positive for $V=y'^2+y^2\ge\frac12$. Thus a solution will move up resp. down the level curves of the squared radius function $V$ when outside the interval $$\frac13<V=r^2=y^2+y'^2<\frac12.$$
That the claim follows this easily from the modified task hints strongly that this is the original version and the one in the question the erroneous one.
The code for the first picture is