Consider the equations:
$ \frac{dx}{dt} = y$ and $ \frac{dy}{dt} = -x $
By transforming variables, obtain $ \frac{dr^2}{dt}=0 $ and $ \frac{d \theta}{dt} = -1$
I know that if I say let $r^2 =x^2 +y^2 \cdot \frac{dr^2}{dt} = tx \frac{2x}{dt} + 2y \frac{dy}{dt} =2xy =2xy-2xy = 0$, which would mean $r^2$ is neutrally stable, but I don't understand where this gets me, much less what a change of variables is.
Moreover, what can one conclude about whether these are circular solutions, their direction, and their stability. I have not seen this mentioned before.
Assuming you're using polar coordinates, a change of variables to the polar coordinates system corresponds to $x=r\cosθ,y=r\sin θ,r=x^2+y^2$ :
It is :
$$r^2 = x^2 + y^2 \Rightarrow 2rr' = 2xx' + 2yy' \Rightarrow rr' = xx' + yy' $$
Substituting $x',y'$ from your given system :
$$rr'= xy - yx = 0$$
To find the angle $θ$, we take :
$$\dfrac{r \sin \theta}{r \cos \theta} = \tan \theta = \dfrac{y}{x}$$
Using the quotient rule, we get :
$$\theta' = \dfrac{x y' - y x'}{r^2}$$
Substituting $x',y'$ as before, from your given system :
$$θ = \frac{-x^2 -y^2}{r^2} =\frac{-(x^2 + y^2)}{r^2}=\frac{-r^2}{r^2}=-1$$
This is how you get the expressions.
Going over stability, it's easy to see that $O(0,0)$ is a stationary point of your given system and also the only one.
Since you have $rr' = 0$, you can conclude that $O(0,0)$ is clarified as a center for your system.
You can easily double check that by doing the common stability way :
$$J(x,y) = \begin{bmatrix} 0 & 1 \\ -1 & 0 \end{bmatrix}$$
Which means that :
$$J(0,0) = \begin{bmatrix} 0 & 1 \\ -1 & 0 \end{bmatrix}$$
and that :
$$\det(J(0,0) - λI) = 0 \Leftrightarrow \cdots \Leftrightarrow λ^2 + 1=0 \Leftrightarrow λ = \pm i$$
Since you have purely imaginary eigenvalues, $O(0,0)$ is a center for your system, thus we have double checked our finding.
Can you now make a conclusion about the direction and the circularity of the solutions ?