Soft question. I'm taking an introductory mini-course in dynamical systems, and the professor defined a continuous dynamical system in a topological space $M $ (or metric space, smooth manifold, or whatever) as a map (continuous, smooth, etc) as a map $\Phi:M\times \Bbb R \to M$ satisfying
(i) $\Phi(\cdot,0)={\rm Id}_M$ and
(ii) $\Phi(\Phi(x,t),s)=\Phi(x,t+s)$ for all $x\in M$ and $t,s\in \Bbb R$.
He also defined a discrete dynamical system as same as above with $\Bbb Z$ instead of $\Bbb R$, and made a remark saying that we could use a group instead of $\Bbb R$ or $\Bbb Z$, which I guess it would be like this: let $G$ be a group and ask $\Phi:M\times G\to M$ to satisfy
(i') $\Phi(\cdot,e_G) ={\rm Id}_M$ and
(ii') $\Phi(\Phi(x,g),h)=\Phi(x,g\cdot h)$ for all $x\in M$ and $g,h\in G$.
Although we'd probably want $G$ to be a nice topological group or Lie group (we could start talking about continuity or smoothness of $\Phi$), what I'd like to know is: what interesting stuff can be modeled using a dynamical system like this? I'm curious about the usefulness of non-abelian groups in this context, since we'd probably want the parameter space $G$ to represent instants in time.
What do you mean by "modeled"? If you want to keep the idea of time, I think there is little "modeling" in that sense for non-abelian groups (I'll stand corrected if some example arises). If by "model" you mean applications to physics, the notion of an action of a group on a given manifold, for example, has intrinsic relations and usefulness to the notion of symmetry (which is obviously invaluable in physics, and where motivated non-abelian groups arise naturally - $\mathrm{SO}(3)$, for example). In this sense, the dynamical system can be interpreted as there being a way to move around the space via certain specific movements which are of interest to the specific case and which behave in "concatenable" ways. Implicitly, this is also what it is really happening when you are given an action of $\mathbb{R}$ or $\mathbb{Z}$: the way of moving being flowing forwards/backwards on time.
If you want to know the usefulness of the notion (in mathematics itself, say) in perhaps surprising situations, then there are infinite. For example, an elementary usefulness is knowing that $\mathrm{SO}(n)$ (non-abelian) acts on the sphere $S^{n-1}$ with isotropy "equal to" $\mathrm{SO}(n-1)$, and hence $\mathrm{SO}(n)/\mathrm{SO}(n-1) \simeq S^{n-1}$. This allows one to conclude rather cleanly, for instance, that $\mathrm{SO}(n)$ is path-connected for every $n$ (there is a result which says that if $G/H$ is connected and $H$ is connected then $G$ is connected).
More interestingly, one can give a counter-example to the "pseudo-Poincaré conjecture" that a compact manifold with the homology of a sphere is homeomorphic to the sphere. This is done by using the fact that the quotient of a properly discontinuous action of a group $G $ on a simply-connected manifold $M$ has fundamental group equal to $G$ itself, which is a good application by itself (for example, $\mathbb{Z}\oplus \mathbb{Z}$ acts on $\mathbb{R}^2$ by translation, with quotient being the torus, which allows us to conclude that $\pi_1(T^2)=\mathbb{Z}\oplus \mathbb{Z}$. Another example is the action of $\mathbb{Z}/2\mathbb{Z}$ on the sphere which furnishes the projective space). Omitting details, one constructs a properly discontinuous action of a perfect $G$ group on $S^3$. By the theorem I mentioned, the fundamental group of the space $X$ given by the quotient of the action is necessarily $G$. In particular, it can't be homeomorphic to $S^3$. However, by Hurewicz, $H_1(X)=0$, and by duality $H_2(X)=0$ (since $H^1(X)=0$ by the universal coefficient's theorem).