I am trying to show that the origin is the only critical point of $$x'=y+kx(x^2+y^2), \ \ y'=-x+ky(x^2+y^2) \ \ \ \ k\in\mathbb{R}.$$
If we re-write in polar coordinates, we can show that $$r'=kr^3.$$ Then $r'=0$ is only satisfied when $r=0\implies x=y=0\implies x'=y'=0$.
The answer in my textbook used a much more rigorous algebraic proof (i.e. manipulate both $x'$ and $y'$ equations to find that $x=y=0$ is the only solution which satisfies $x'=y'=0$). Is my method legitimate?
Yes your method is correct.
$$x'=y+kx(x^2+y^2), \ \ y'=-x+ky(x^2+y^2) \ \ \ \ k\in\mathbb{R}.$$
Note that $$rr'= xx'+yy'$$
$$rr'= xy+kx^2(x^2+y^2)-xy+ky^2(x^2+y^2)=k(x^2+y^2)^2$$
which does imply to $$r'=kr^3$$
The origin is the only equilibrium.