The system is $$\dot{x}=-y+ax(x^2+y^2)$$ $$\dot{y}=x+ay(x^2+y^2)$$
and the variables are changed to $x=r\cos\theta$, $y=r\sin\theta$,
and when you note $x^2+y^2=r^2$,
why can you say $x\dot{x}+y\dot{y}=r\dot{r}$ ?
The system is $$\dot{x}=-y+ax(x^2+y^2)$$ $$\dot{y}=x+ay(x^2+y^2)$$
and the variables are changed to $x=r\cos\theta$, $y=r\sin\theta$,
and when you note $x^2+y^2=r^2$,
why can you say $x\dot{x}+y\dot{y}=r\dot{r}$ ?
This is implicit differentiation with respect to a variable (which hasn't been named). As $x$, $y$, and $r$ are all functions of this variable which I'll call $t$ (since it's presumably time if this is a physics problem), you could think of the equation giving their relationship as
$$ x(t)^2 + y(t)^2 = r(t)^2. $$
Differentiating this with respect to $t$ gives (chain rule)
$$ 2x(t)x'(t) + 2y(t)y'(t) = 2r(t)r'(t) $$ and we can divide out all the $2$s. In practice, writing "of $t$" over and over is a pain, so we usually write it the way you did in the problem instead (nothing is lost as long as these are all functions of just one variable).