To fix ideas: let $Q$ be a set of states and $\Sigma$ a set of inputs; let $\delta : Q \times \Sigma \rightarrow Q$ be a transition function. If $Q$ and $\Sigma$ are finite, $\delta$ in conjunction with an initial state $q_0$ and an accepting set $F$ specify a discrete finite automaton.
If $Q$ and $\Sigma$ are continuous, what is the analogue of a DFA called (I am guessing this is some flavor of a "hybrid system" that crops up in the electrical engineering community)? In particular, are there any nontrivial results known for "nice" $\delta$ (e.g., convex, known qualitative properties of derivatives, etc.)?
Your question is rather unclear: what do you mean by "If $Q$ and $\Sigma$ are continuous"? Anyway, the closest thing I can think of is a topological transformation semigroup, that is, a transformation semigroup $(Q, S)$ in which $Q$ is a topological space, $S$ is a topological semigroup and the action $Q \times S \to Q$, $(q, s) \to qs$ is continuous. This notion is used for instance in [1], but this is just the first reference I found on Google and there are certainly other sources. In a dynamical system, the definition is the same, but the semigroup is usually commutative.
[1] Haynes, Tyler. Thickness in topological transformation semigroups Internat. J. Math. Math. Sci. 16 (1993), no. 3, 493--502.