I've researched a lot trying to do this question. The only way I can think of solving this problem is using a differential equation. But it seems this question requires a discrete map to solve it. Can anyone help?
"Consider the set of points on a circle of radius r about the origin and show that they are mapped under one step of the dynamical system to another circle which you should specify."
Your map is $z^2+(a,b)$ and so the circle of radius $r$ centered at $0$ is taken to the circle of radius $r^2$ centered at $(a,b)$. Alternatively you can write $$ (f(x,y)-a)^2+(g(x,y)-b)^2=(x^2-y^2)^2+(2xy)^2=(x^2+y^2)^2=r^4 $$ to conclude the same.